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ABSTRACT 

Research on the implementation of the Job Training 
Partnership Act (JTPA) included a sample of more than 20,000 people 
in 16 service delivery arear (SDAs). The random assignment model for 
the study produced estimates of the impact of the overall JTPA 
program in this sample of SDAs, plus separate impact estimates for 
individuals recommended for classroom occupational training and 
on-the-job training. Because some SDAs declined to participate due to 
fears that the study would disrupt or have a later negative impact on 
their operations^ the U.S. Department of Labor and the research team 
altered the exclusion rules for the control group, provided technical 
assistance to help SDAs and service providers meet enrollment goals, 
simplified the research design, increased the compensation for sites • 
efforts, and urged states to administer the performance standards 
system flexibly. The following lessons for future research were 
developed! (1) when research goals conflict, priorities must be set 
quickly so that a realistic plan be developed; (2) if 
generalizable findings are necessary, major adjustments may have to 
be made in the study design to recruit a representative Sdmple; (3) 
if participation by sites is voluntary, sites must be offered 
incentives to participate; (4) implementing a random assignment 
research design changes the operation of an ongoing program; and (5) 
technical assistance and staff training are important. (Appendices 
provide an overview of JTPA, profiles of the sites, tables comparing 
participating SDAs with others not participating; and 45 references.) 
(KC) 
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PREFACE 



In some respects, this is an unusual report. TTie project it describes - the NaUonal 
JTtA (Job Training Partnership Act) Study - is ambitious, going beyond what has been 
ettempted in past research. But the report is unusual in another way: It provides a detaUed 
analysis of the "reconnaissance" preceding the start of a major study and ends where initial 
reports on projects often begin, by characterizing the research sample for the study. 

The basic purpose of the report is to describe how the research team identified 
tradeoff among the iniUai goals for the project and resolved them by adjusting the research 
plan's site selection and research procedures. While f :nising on the National JTPA Study, 
the report draws a number of general lessons for researchers and policymakere concerned with 
social programs, especially the necessity of creating consonance between the goals of a study 
and the operational realities of the program. TTiese lessons are summarized m the final 
chapter. 

The process of developmg the research plan and selecting sites was demandmg because 
of a combination of three factors: the study's original goals, the character of the JTPA system, 
and the research method. The U.S. Department of Labor, which instituted the study, originally 
sought to evaluate JTPA in a randomly selected group of sites, without changing normal 
program operations. The goal was to achieve estimates of the unpact of individual types of 
activicies in addition to estimates of the overall impact of the local programs. Accomplishing 
these goals was complicated by the structure of JTPA - a highly decentralized, voluntary system 
offering a diversity of employment and training services. Local and, to a lesser extent, state 
officials play a central role in settmg program priorities, and program services are continuously 
assessed by a performance standards system which, during the period covered by this report, 
emphasized high rates of job placement and low costs. For the reasons discussed in the first 
chapter of this report, the Department of Labor chose to use a classical experimental design, 
in which apf icants are randomly assigned to either an experimental group (eligible to receive 
JTPA services) or a control group (eligible to receive only non-JTPA services in the 
commum*ty). This report is largely a discussion of how these three factors interacted and how, 
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through a lengthy process of adjustments, the sites came to join the study under a research 
design that retained most - but not aU of the original goals of the study. 

Judith M. Gueron 
President 
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EXECUTIVE SUMMARY 

Each year, programs funded under the Job Training Partnership Act (JTPA) of 1982 
ofifer classroom vocaUonal training, on-the-job training, job search assistance, and other services 
to nearly one milhon economically disadvantaged youth and adults. They are served through 
approximately 600 local JITA agencies called service delivery areas or SDAs, which annually 
receive some $1.8 billion in federal funds under TlUe EA, the largest part of the AcL 

In common with aU employment and training programs, JTPA aims to move jobless 
people into jobs. But reflecting criUcism of past programs, the JTPA legislation put special 
emphasis on placing people into private sector jobs, containing costs, and assuring program 
accountability. Hie Act also required the Department of Labor - the federal agency 
overseeing JTPA - to conduct a study of how well services funded under the program were 
achieving their central objecUves: increasing employment and earnings and reducing welfare 
receipt The National JTPA Study is the department's major response to this charge; its 
subject is the impact of JTTA on adults and out-of-school youth served under Title DA. ITiis 
report is about the implementation of the study and the lessons that can be drawn for other 
research. 

BACKGROUND OF THE STimv 

To understand the impact of employment and training programs, policymakers and 
program operators seek information on how weU program participants fare in the labor market 
over and above what they would have done on their own, without the program. In other 
words, they need to know what difference a program makes. 

Evaluation experts and poUcymakers have increasingly recognized that the most reUable 
studies of program impacts use random assignment - a process identical to a drawing, m which 
eligible applicants for a program are randon..y assigned to a group to be served (the 
"treatment- or "experimental" group) or to a group not given access to the program (the 
"control" group). Because the two groups are created randomly, there is no systematic 
difference between them prior to random assignment This means that the labor market 
experiences of the control group will provide an accurate benchmark for how the treatment 
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group ivould have done .vithout the program. Hen«, by cmpaiing the differences between 
the two groups - for example, in leveU of earnings or welfans receipt - one can measure true 
program accomplishments and the diffensnce the program has made. 

Since 1975. random assignment «udies have gained vndc acceptance in the research 
commmuty as the most reliable way of measuring the effectiveness of employment and trainmg 
programs and, for this reason, the Department of Labor chose this method for the NaUonal 
nPA Study. However, the department recognized that the task in the previous random 
assigmnent research was different from and in many waj. simpler than that faced in the jn>A 
study. 

Virtually aU the past random assigmnent research analyzed nev;- programs (some 
estabhshed as part of a demonstraUon); the JIPA study examines an established sj.tem of 
services. Most past studies analyzed a specific program model, with services offered through 
a smgle agency working, in some instances, with a few outside service provider. JTPA includes 
a wide variety of employment and training services offered by an often large and varied group 
of providers. TUus, the JIPA study requires complex coordinaUon to negoUate detafled study 
procedlures and to assure that the study is being properly implemented - notably, that the 
members of the control group are not receiving JTPA-funded sei>.-ces anywhere in the local 
area. In most past random assigmnent research, only one type of client was targeted: welfare 
recipients or school dropouts, for example. Under Title HA of JIPA the clientele is far more 
dive«e. including youth, adults, long-term welfare i^edpients. newly unemployed people, older 
workers, handicapped individuals, and others. 

The quesUons to be answered are also more complex than in many past studies: In 
addition to the usual overall assessment of a program, the study calls tor separate impact 
estmiates for specific types of activiUes and parUcular groups of people within the sample. 
Fmally. though sites were not required to participate in the study, the department sought to 
recruit a statisUcally representative sample of local programs in order to be able to generalize 
study findiiigs to the entire JTPA system. 

The character of the program or the nature of the research questions alone would have 
presea.ed unusual challenges. Combined, they make the National JIPA Study a umque 
challenge. 
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IMPLEMENTING THE STUDY DESIGN 

Despite these ambitious jjoals, the department and research team are now implementuig 
a study design that will accomplish most of the project*s original objectives. The final research 
design includes a sample of more than 20,000 people in 16 SDAs, The random assignment 
model for the study will produce estimates of the impact of the overall JIPA program in this 
sample of SDAs, plus separate impact estimates for individuals recommended for classroom 
occupational training and on-the-job training. Program impacts will be calculated for adult 
males, adult females, and out-of-school youth. Information on the employment, earnings, 
welfare receipt, and other experiences of members of the research sample will be collected for 
approximately 30 months following random assignment^ 

One objective of the original study plan has not been achieved: recruitment of a 
statistically representative sample of sites. From the beginning of the project, it was understood 
that this would be difEcult and time-consuming. In most SDAs, the SDA professional staff, 
the Private Industry Council, service providers, and local elected officials ail weigh in on 
important decisions. Thus, people with widely varying perspectives had to be convinced that 
the study would not cause serious problems before an SDA would agree to participate. Many 
sites targeted for selection did not join the study because of qualms about how the research 
design would affect their operations. 

The key reasons SDAs declined to participate included (1) misgivings about the nature 
of random assignment research and the study's effects on whom they might enroll, (2) problems 
meeting enrollment goals, which would intensify because one-third of their applicants would be 
diverted to the control group, (3) administrative burdens from the study, which they perceived 
as exceeding the study*s benefits to them, and (4) apprehension that participating in the study 
might hamper their ability to meet JTPA performance standards (used to award special 
incentive funds) or might disrupt existing performance-based contracts with their service 
providers. 

This report describes the responses developed to address these and other concerns as 
part of an iterative process of modifying the research design and consulting potential sites. To 



'The National JIPA Study has two additional components: (1) a special study of the process of 
recruitment and sclcaion of panicipanis into JTPA and (2) research usmg the random assignmeni Held 
study as an opportunity to improve other methods of csiimatmg program impacts. 
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summarize the changes made: The dcjparteem and reseaidi team altered the oclusion niles 
for the control group, provided technical assistance to help SDAs and service providers meet 
cmolhnent goals, amplified the research design somewhat to ease the administrative burden, 
increased the compensation for sites' efforts, and urged states to administer the performance 
stacdards system flexibly. 

Eventually, it was possible to recruit 16 sites that reflect the diversity of the JIPA system 
along a number of dimensions: region, size, ethm'c composition, population density, and JTPA 
performance ratings. Hence the study wiU ascertain the impact of programs in a variety of 
local settings and for different types of clients. Nevertheless, tiie effort was a difOcult one. 
and many sites chose not to participate. 

LESSONS F OR FUTIIPF RESEARCH 

Ihis experience highlights a number of valuable lessons for future studies, including the 
foUowing: 

. L V-ljen it b€a)mes clear that research goals conflict, priorities oust quickly 
fee set so that a reaUstic plan can be developed. In tiie JIPA study, tiie 

.w!irf°\?°^ ^- °^ estimates of pr^ram 

mipacte for the sites m tne study (both overall and for specific treatmfnts) 
^rthe goal of a strictiy representative sample of sites. In otiier studies 
different choices might be made, but early determination of research 
pnonties can conserve time and resources. 

^ ^eeaeralizable findings are the highest priority, mzjor adjustments in the 
shidy des^ may be needed to recruit a statistically representative sample 
of sites. Possible adjustinents in the JTPA study might have included an 
easier-to-miplement random assignment design tiiat answers fewer questions 
greater financial support for participating sites, and mandatory partidpatior; 
for selected sites. All tiiese solutions pose tradeoffs witii other research 

3. If participation by sites is voluntary, sites must be offered an appropriate 
package of fina^-ir; and nonfinapxi.! benefits. Since the major impact 
findings for tiie entire sample will be available to all programs whether they 
participate or not, other inducements must be offered. These could include 
sitc-specific impact findings, additional program funds, and opportunities for 
mvolvement m the national poUcy debate. Costs such as added administra- 
tive effort, the risk of performance standards problems, and operational 
disruption must also be addressed. 
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4. Impleisentiisg a cempler random assigmnenti^eseai^desi^ 

program inevitabfy dbanges its operation somewhat Researchers should 
identify ^vhich aspects of the program it is essential to protect as much as 
possible &om changes. They must then develop a research plan that leaves 
those elements virtually intact In the JTPA context, prime candidates to 
retain with as. few charges as possible are assessment procedures, services 
offered, and performance standards. Other situations might require other 
choices* 

5. Technical assistance and careful training of site staff are important A 
stucfy such as this affects the activities of many staff members of SDAs and 
service providers; all must receive training on the pur^^ses and procedures 
of the study. The study's implementation can also be aided by providing 
technical assistance to sites on key issues such as client recruiting. 

These lessons and others are discussed in detail in Chapter 6 of the report 
MDRC staff responsible for launching the National JTPA Study concluded that it is 
possible to conduct a large-scale, random assignment field study within a system such as JTPA. 
But they also found that implementing this study's research design in ongoing programs 
operating under performance standards and facing difficulties recruiting appropriate clients 
pushed random assignment research to the limits of its feasibility. Future reports on this 
project will document the nature of JTPA services provided by study sites, analyze their 
impacts, and explore new methods for conductmg this type of research. 
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CHAPTERl 

THE PURPO SES OF THE N ATIONAL TTPA STimv 



The Job Training Partaenihip Act of 1982 (JTPA) is the nation's largest employment and 
training program for disadvantaged adults and youth. It authorizes federal grants to states and, 
through them, to approximately 600 local agencies (caUed service delivery areas or SDAs), which 
provide classroom vocational training, on-the-job training, job search assistance, and other services 
to more than one miUion people a year. 

JTPA marks an evolutionary step in program design. Its predecessor, the Comprehensive 
Employment and Training Act (CETA), had drawn criticism for not placing more people in 
unsubsidized, private sector jobs; high costs; and insufGcient accountability. Congress responded 
by focusing JTPA on private sector employment and by requiring the Department of Labor to 
monitor how weU the new program meets that objective. The Act also requires the department 
to conduct a study of the program's s.uccess in achieving its statutory goals: increasing employment 
and earnings and reducing welfare dependence. 

Over tiie last decade, methods of evaluating employment and training programs have also 
evolved. During the CETA years, the Department of Labor funded the Continuous Longitudinal 
Manpower Survey (CLMS) as weU as extensive research that used the survey's data to asse<c the 
p.-ogram. Unfortunately, despite the use of advanced statistical techmques, the CETA arch 
did not identify an appropriate comparison group against which to measure the program * effects 
on participants. TTius, the findings were inconsistent and confusing on the key question: the 
program's impact on participants' employment and earnings. For JTPA the department chose 
a different approach: a random assignment Oeld study of JTPA in a sample of local programs. 
Those unfamiliar with this research approach will find many of its key principles discussed in 
Chapter 4. 

This report describes the early implementation of the National JTPA Study, which was 
undertaken by the Department of Labor to satisfy the Congressional mandate to study JTPA's 
effectiveness. The report explaits how tradeoffs among research objectives were balanced in 
developing a detailed research desifei acceptable to sites; describes issues that arose in site 
selection; compares the sites in the study th- national JTPA system, and presents some lessons 
for other research. It covers events through the end of random assignment in September 1989 
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and does not include findings about the impact of the program. 

This study, being conducted in 16 sites across the country, with a sample of more than 
.20,000 persons, represents something new in social policy lesearch. Whfle random assignment 
has been used to study spedal demonstraUon? and smaller employment and training programs, 
this study is the first time butside the welfare context that it has been used to assess the impacts 
of an ongoing, large^e employment and training program. 

nie National JTPA Study is a timely effort, coming when there is intense interest in 
employment and training programs in general and JTPA in particular. There is concern in both 
the public and private sectors that many entry-Ievel workers may lack the skills needed in our 
increasingly service-oriented economy, a concern heightened by demographic projections that the 
labor ibrce tviU grow more slowly from 1985 to 2000 than in any other period since the 1930s.^ 
These labor market issues have also thrown a spotlight on those people outside the economic 
mainstream, with littie work experience and low skills, leading to new calls for programs to address 
their needs and prepare them for employment 

Attention has naturally focused en JTPA, and espedaUy on Tide HA of the Act as the 
largest source of funds for employment and training programs Under Title HA, approximately 
$1.8 biUion of federal aid is distributed annually, a level of funding that permifs the programs to 
serve cjly 5 to 10 percent of the eligible population.^ Chapter 2 and Appendix A of this report 
present more background on JTPA for those not familiar with the program. 

This chapter presents the goals of the National JTPA Study. It then reviews why the 
department dedded to use a random assignment field study for assessing JTPA's impacts, 
underscoring how this project moves beyond previous random assignment research. Finally, the 
chapter summarizes the research plan and describes the organization of the report 

I- The Researc h Goals of fhe National JTPA Study 

Section 454 of the Job Training Partnership Act directs the Department of Labor to study 
the effectiveness of programs authorized under JTPA in achieving three goals: increasing 



'Fullcrton. 1987. 

2 Toial funding for all pans of J7PA has been approximately $3.7 billion in recent years. The 
attention directed at JTPA is illustrated by the formation and subsojuent reports of the JTPA Advisorv 
C^mittee to the U.S. Seacuiy of I^bor. See Job Training Partnership Aa (JTPA) Advisory Committee' 
1989. In addition. Congress is currently considering several bills that would significantly amend JTPA. * 
Gnnker Associates, 1986; Cook ct aU 1985; Walker, Feldstein. and Solow. 1985. 
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participants* employment and earnings, reducing income support costs, and increasing tax revenues. 
This statutory language calls for an analysis of program impacts not post-program outcomes^ a 
crucial distinction for understanding the study's research goals and approach. 

Basically, the statute and the National JTPA Study seek to learn what difference JlPA 
makes in the lives of those who participate in its programs: Do JTPA services result in levels of 
employment and earnings that are higher than they would have been without the services? To 
assess this (the program impacts), the department cannot merely observe the post-program 
employment rates, earnings, and rates of receipt of welfare (the program outcomes) for people 
who participate in JTPA'^ Program outcomes are not measures of the difference JTPA makes 
because some of those served would have found a new job on their own or improved their skills 
and raised their income through other means even if they had not participated in the program. 

Thus, crediting all post-program "success stories" to JTPA overestimates the program's 
impacts. In fact, if a program served only very employable people who could find a job on their 
own, all the participants might be employed after the program and yet the program itself might 
have had no impact on their employment 

The starting point for assessing the difference a program makes is a measure of what would 
have happened to people if they had not had access to program services. This provides a baseline 
against which to compare the program's accomplishments. The measure of a program*s impact 
is the difference between what would have occurred without the program and what actually 
occurred following participation in the program.^ As discussed shortly, this calls foi creating and 
studying groups of people who are identical (or nearly so) except for access to the program: a 
control group (with no access to the program) and an experimental or treatment group (with 
access to the program). 

Responding to the statutory directive for an impact study of JTPA, the department issued 
a Request for Proposals (RFP) in early 1986 to conduct a random assignment field study of Title 
HA in up to 20 SDAs. In the RFP, the departraer: luentified three major research goals for the 
study: 



^csc types of outcome measures are used in the existing performance sundards system, v^hich plays 
a central role in JTPA management Performance sundards are discussed in Chapter 1 

distinction between outcomes and impacts is not the same as the distinction between short-ienr. 
and long-term measures. Even long-term measures of program outcomes only measure the post-program 
status of the group served, not the difference the service made in their lives. 
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• Goal#l: Ettimateboth (a) UwoveraUlmpactof jmTitkIIA»c«^^^ 
provided in «s(A of a sampk of local prognuns and (b) the spedffc impact of 
important catesories of activities such as on-the-iob training and classroom 
occnpational traialng. 

An overaU impact estimate is important for addressing Congressional interests, as expressed 
in the statutory requirement to assess the effectiveness of JTPA activiUes. Activity-specific 
estimates are important for poHcymakere seeking to learn what types of employment and training 
activities to fund and to program planners making decisions at the local level about service 
contracts. 

Goal #2: Estimate program impacts for important subgroups such as adult 
men and women, and minority and white out-of-school youth. 

TTiis approach was adopted for two major reasons. Hrst, previous studies identified 
subgroups of participants for whom programs appeared to work differentiy; the JTPA study was 
designed to determine whether these patterns persisted when JTPA was studied with a more 
reUable research method than those used in the past Past research suggested, for example, that 
traming programs tended to have larger impacts on the earnings of women than of men.^ It also 
found that youths' experiences in the labor market were quite different from adults' and that 
minority youth had special problems finding and keeping stable, well-paying jobs."' Second, JTPA 
requires local program operators to "provide employment and training opportum'des to those who 

can benefit from, and who are most in need of; such opportunities The study made it 

possible to learn more about the overiap between groups most in need of assistance and groups 
that benefit from services. 

In-school youth served under TiUe HA, who constitute 10 to 15 percent of Title HA 
participants, are excluded from the study for two reasons: 

a. In-school JTPA programs usually have very different goals in the short term 
than do programs for out-of-school youth and adults. Generally, the former 
do not seek placement in a job as an immediate goal, while the latter do. Thus, 
given the study's desired focus on more immediate impacts on employment and 
earnings, the in-school group was not as central, 'ftirthermore, its incliision 
would have required a separate research design. 



'See Bamow. 1987. 

'See. for example, Ostcrman, 1980; Freeman and Wise, 1981 
^Section 141(8) of the AO. 
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b. It would have been far more difBcuIt to implement the study procedura in a 
school setting, as will become apparent when the research approach is outlined 
later in this report In a school setting, special problems would have arisen 
about denying JIPA services to the control group and about mom'toring and 
enforcing the intended differences in services between those randomly assigned 
to the program and the control group. 

• Goal #3: Use the opportunity of a random assignment field study to seek 
better ways to Identify an appropriate comparison group through means other 
than random assignment 

The department recognized that a random assignment Oeld study is a major undertakmg that 
is difficult to conduct on a routine basis. The REP caUed for quasi-experimental^ analysis to 
seek other ways to measure the average outcomes that JTPA participants would experience 
without JTPA services. 



n. The Use of Random Assignment fn the National TTPA Study 

The department's choice of random assignment for this study came after an intense 
examination of two alternative research methods to determine program impacts: 

1. Nonexperimental approaches that IdenUfy a matched comparison group similar 
to program participants. 

Members of the comparison group would be individuals who appear similar to participants 
on measurable characteristics such as age, education, sex, race, work history, and previous 
earnings. The comparison group might be chosen from people who applied for the program but 
were never admitted or who never applied but were eUgible for its services. The behavior of 
participants and the comparison group is compared over time. 

2. Random assignment to a "treatment" group or a "control" group that Is not 
to receive the program's services. 

Random sssignment is a lottery-like process in which large numbers of applicants for a 
program are assigned to a "treatment" group, which is given access to the program, or a "control" 



'^ndom assignment Impan research is often called "•expcrimcnul* rese<irch because u was onginall-. 
used in classical, often medical or agricultural, experiments assessing the impan of different •treatments.'' 
The method is now also used in the behavioral and social sciences. Non-random assignment impan 
research, in oontiast is often called nonexperimental or quasi-cxpcrimenul research (sec Achen, 
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group, .diich is not given access to the program. Because the two groups arc created through 
a random process, there are no systematic differences between the prior ciperience and 
characteristics of the two groups. Thus, a comparison of their employment, earnings, and welfare 
receipt after random assigmnent yields an unbiased assessment of the impact of the program 
under study. 

Both approaches had been used in past studies of employment and training programs. The 
evaluations of the CETA program using the CLMS had relied on a comparison group 
methodology to assess program impacts. Random assignment had been used to study several 
important demonstrations of new program models such as Supported Work and JOBSTART, 
and 6 variety of employment programs for dislocated workers under Tide m of JIPA and for 
recipients of Aid to Families with Dependent Children (AFDC), the major federally funded 
welfare program.JO Using random assign .>ent to create a control group was widely viewed as 
providing more accurate and reliable estir^ate. of program impacts than using statistical matching 
to create a comparison group. In the terminology of the evaluation literature, random assigmnent 
research - properly designed and implemented - is much more likely to produce "intemaUy valid- 
estimates of program impacts: that is, accurate or "unbiased- answers to the question of whether 
the treatments did make a difference in the speciGc instances studied. 

As wiU become apparent in this report, however, implementing random assigmnent Geld 
studies is a challenge for partidpating sites. In past random assigmnent studies, a considerable 
proportion of local programs contacted about participating have bccu unwilling or unable to do 
so. In coulrast, nonexperimental research, such as fjie work done using the CLMS and 
comparison groups, requires very Uttie daily cooperation from programs in the study and can be 
conducted in large numbers of randomly selected sites. Thus, it is usually more difficult to recruit 
a representative sample of sites m random assignment studies than in research such as that done 
on CETA As a result, the "external vaUdity" or generalizabiUty of the findings in random 
assigmneni studies is more often an issue." However, as the discussion below will indicate, there 
was a growing consensus that random assigmnent is necessary for intemaUy vaUd estimates of 
the impacts of employment and training programs. 

In developing a plan for studying JTPA the department was well aware of this tradeoff. 



11^^ ?h^f^ . ""J^ "^P'"* ""'^y "^"""'"g entire, ongoing program 

study possesses external vaUdity. See Bracht and Glass, 1968: Campbell and Sunlcy 1966 
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It carefully reviewed previous efforts to evaluate employment and training programs to draw 
lessons for the upcoming study. A brief historical account iUustrates the issues the department 
faced. 



A. Past Researvh on Employment and TrainincProigrams 

During the 1960s ijid 1970s, the Department of Labor invested bilUons of dollars in 
programs for adults and youth and funded many evaluations of their effects.^^ evaluation 
of the CETA prograiii was among the largest undertaken during the late 1970s and early 1980s. 
The department funded the CLMS, which collected data on the socioeconomic characteristic, 
program participation, and work histories of a national sample of individuals served by CETA in 
147 local programs. These data were avaflnble for samples of participants for fiscal years 1975 
through 1981. In addition, the CLMS contained data from the March Survey of Current 
Population (CPS) to allow identification of individuals who could make up a comparison group 
for the analysis of program impacts. (March is the month in which annual demographic and labor 
market information is collected.) 

The department funded an im'tial analysis of program impact^:^^ ^^^^ supported 
reanalyses of these data by other researchers (using slightly different techm'ques to identify an 
appropriate comparison group and analyze impacts). Unfortunately, Uiis series of studies, using 
state-of-the-art nonexperimental research techniques, did not produce r consensus on CETA's 
impacts. 

Table 1.1 illustrates the wide range of estimates of CETA's impact on adults' annual 
earnings; the calculations were made over the course of a decade.^^ The initial research using 
the CLMS is labeled Westat (1981). The other studies represented in the table are reanalyses 
of the CLMS data, using slightly different techniques. The most extreme examples of inconsistent 
results concerned adult men: Estimates of the overaU CETA program's impacts on annual 
earnings ranged from $200 to minus $700, and estimates of Uie impact of CETA's on-the-job 
training for white men ranged from a high of $1,000 to $U00 to a low of minus $200. Estimates 



*¥or a review of early research on employment and training programs, see Perrv ei al.. 1975 For 
a review of research on youth programs, sec Betsey et al.. 1985. For a review of the research on the 
CETA program, see Bamow. 1987. 

"Wcsiai, 1981. 

**Tbc estimates in Table 1.1 are from a group of studies that cover slightly different penods of 
program operations. Sec Baniow. 1987, for a review of these studies and a discuss'ion of the differences 
in methodology and coverage. 
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SOURCE: 



NOTES: 



0 



Bamow, 1987. Sources for estfiaates are listed in the references at the end of this report. 
Estinates are for all tdult participants except as otherwise indicated. 

All estimates are In post-progra» year dollars except for Blooai ft McLaughlin estiiMtes, which are in 1980 dollars. 

Hissing entries indicate that iinpact estimates were not calculated. 

Numbers in parentheses ore negative inpact estimates. 

♦Denotes statistical significance at the 5 percent level. 

Estinatet are for all adult participants except as otherwise indicated. 



29 



for other activities and subgroups had narrower ranges but still showed so much uncertaiqty that 
they would have been di£5cult to use for policy planning. Therefore, policymakers did not know 
if CETA actually increased participants* employment and earnings. 

B. Planniog the Research on .TTPA 

The department originally planned to study JTPA using a comparison group approach similar 
to that used for CETA. With the start of JTPA, the department established the Job Training 
Longitudinal Survey (JTLS) to collect data on the socioeconomic characteristics, program 
participation, and work histories of a national sample of adults and youUi served by JTPA 
Moving beyond the CLMS, the department planned to supplement the CPS data witii the Survey 
of Histoiy of Work (SHOW), a specially conducted national survey designed solely to be used in 
constructing a comparison group for studies of JTPA impacts.^^ 

As the department was doing the final planning for this research, the series of inconsistent 
nonexperimental findings on CETA began to emerge. Seeking a consensus in the research 
commum'ty on the best w^ to conduct nonexperimental research on employment and training 
programs, in late 1984 the department appointed a panel of evaluation experts to review its plans 
for studying JTPA and to recommend ways in which to deal wiUi the unresolved meUiodological 
problems. The panel devoted most of its attention to the central methodological problem of 
"selection bias." 

Possible selection bias has continually been an issue in nonexperimental research like that 
done on CETA because of the difficulty of choosing tiie proper comparison group. Researchers 
can only match members of the partidpant group with other people on observed or measured 
characteristics such as income, education, sex, and ethnicity. If the probability of applying for and 
being accepted into a program were affected only by observed characteristics, tiien selection into 
programs could be modeled statistically and could be taken into account in choosing a comparison 
group. But the probability of applying to and being selected for a program is also likely to be 
influenced by characteristics such as personal appearance and demeanor, motivation, or work 
attitudes which are very difficult to measure reb'ably and are not included in most large data 
sources. 



For a detailed discussion of the JTLS and SHOW and a comparison with the CLMS. see Westai. 
September 1984. As part of the JTLS, the department also l)cgan the Quicjc-Tumaround (Qi") data 
collcaion. in which JTPA enrollment, parudpation.and termination informatic.n is colleaed quanerl> from 
administrative records for a sample of JTPA cnrollees and icrminecs. 



In estimating program impacts, selection bias occurs if imc^>seived diaracteristics affect the 
likdibood of participation in a program and also affect later csqperiences. In a simple and 
potentially important case, individuals vAxo are highly motivated to succeed could be more likefy 
to participate in a program and to have high earnings in the future whether or not th^ 
pmtiqpated in a program. In this case, participant and comparison groups matched on oteerved 
characteristics are different in two ways rather than one: The participant group has access to the 
program and it includes more motivated individuals. If the participant group is later observed to 
have higher earnings, that difference could have arisen because of its greater motivation as well 
as or instead of its participaUon in the program. Therefore, a comparison of the earnings of the 
two groups could overestimate the impact of the program.^^ 

After reviewing the CETA research record and the statistical and econometric techmques 
available to address the selection bias problem, the panel concluded: 

that the estimates of the net impact of CETA are not reliable and that the true 
impacts of CETA are still open to question. Since the methods intended to be used 
to evalua te th e Job Training Partnership Act are broadly the same as those used to 
evaluate CETA, there is considerable likelihood that the vaUdity of the net impact 
evaluation of JIPA will also be subject to unresohrable doubt if the existing analysis 
design remains unchanged^^ 

Hie panel found it veiy unlikefy that any method of addressmg the problem of selection bias 
would be found in time to conduct a vaUd unpad study of JTPA: 

The recommendations of the panel are strongly conditioned by the judgement that it 
will not be possible to soWe the problem of selection bias within the context of a 
quasi-experimental design such as the JTLS/SHOW; at least, not in a short enough 
time frame to meet Congress' needs for valid information to guide poUcy. . . . Even 
though many authors studying employment and training programs have recognized the 
selection problem, no such study using a quasi-experimental design can be said to have 
controlled adequately for selection bias. The panel does not intend to set forth a 
counsel of despair. Rather, it is concerned that the past evaluations of CETA have 
consumed, and the contemplated evaluations of JTPA will consume, millions of dollars 
and much valuable time. It would be extremely unfortunate if the analysis of 
JTLS/SHOW design would yield tiie same ambiguous conclusions as has the analysis 
of the CLMS/CPS data base for CETPJ^ 



^^corciical approaches cxisi for siaiisiically corrcaing ihcse differences, bui ihey often fail lo 
provide reliable and ccnsisicni csiimaies because of ihcir sensuiviiics lo siaiisucal assumptions. See 
Heckman and Robb. 1985. for a review of ihese approaches. 

^^Job Training Longiiudinal Survey Researdi Advisory Panel 1985, p. 2. 

^^Job Training Longiiudmal Survey Research Advisory Panel, 19S5, p. 21. 
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In considering the alternative of random assignment, the panel recognized a tradeoE 
Random assignment research, properly conducted, provides the best opportunity to obtain accurate 
impact estimates for the local programs included in the study (Le^ it maximizes the "internal 
vaUdity" of the estimates). But the challenge of implementing random assignment in an ongoing 
program such as JTPA means that some sites that have been selected for the study will not want 
or be able to participate. Thus, it would be difBcult to include a representative group of 
programs to allow generalization to the entire JlPA program (i.e., to establish the "external 
vaKdity" of the estimates). 

Balancing these factors, the panel felt that without internal validity nothing could be learned 
about program impacts. It therefore recommended that the department abandon the planned 
comparison based on the JTLS and SHOW and instead conduct a series of classical experimental 
studies of the impacts of JTPA for selected target groups and activities in a limited number of 
SDAsP 

The National Academy of Sciences, in a separate review of research on employment and 
training programs for youth, reached a similar conclusion in 1985.^ Finding that comparison 
groups in past research often differed markedly from participant groups, the academy's Committee 
on Youth Employment Programs concluded that: 

control groups created by random assignment yield research findings about employment 
and training programs that are far less biased than results based on any other method. 
. . . Future advances in field research on the efficacy of employment and training 
programs will require a more conscious commitment to research strategies using 
random assignment.^^ 

After reviewing these recommendations, the department decided to suspend the 
JTLS/SHOW comparison group research and proceed with a set of classical field experiments. 
The SHOW was not implemented, and the longitudinal data collection planned as part of the 
JTLS was not done. However, the JTLS continues to collect quarterly reports from a sample of 
SDAs on the characteristics of chents and the services they receive. 

In mid.1986, the department announced that it would conduct a random assignment Celd 
study of JTPA to include up to 20 local service delivery areas. It also announced a parallel effort 



Mob Training Longitudinal Survey Research Advisory Panel. 1985. pp. 21-26. 

2°Bcts<7 ct aL. 1985. 

^^Bctsey et aL. 1985, pp. 18, 30. 



to improve none^rimental tedmiques. A special survey would be conducted of individuak ^o 
wre eligible Cdf but did not apply to JIPA in those SDAs participating in the National JTPA 
Study. Data fiom this survey and other cjosting data such as the CPS acd Survey of Income and 
Program Participation (SIPP) would be used in new efiforts to develop a metiiod to construct a 
comparison group simflar to the control group created through random assigmnent Supporting 
this effort would be an analysis of the process by which individuals are recruited and selected for 
JIPA participation in SDAs in the s'.udy. 

The department utilized an unusual contractual and management structure for the project: 

' ff< A of tte Project After a competitive bid, tiie department selected tiie 
Manpowr Demonstration Research Corporation (MDRC) - witii Abt Associates 
as a subcontractor - as Uie Part A contractor to recruit sites, implement tiie 
study m the field, and document tiie nature and cost of tiie services offered by 
participatmg SDAs. ^ 

Part B of the Project- After a separate competi-tive bid, tiie department 
selected Abt Associates - witii MDRC, ICF. and NORC as subcontractors - 
as Uie Part B contractor responsible for developing tiie research design, 
coUecting baseline and foUow-up data on tiie members of tiie research sample 
and conductmg experimental and nonexperimental analyses of program impacts. 

ID- The UoDsual Challenges Ppsed bv Thk Pmf prf 

The National JTPA Study, of course, is not tlie first time random assignment has been used 
to assess tiie impact of federally funded employment and training programs. In fact, its successful 
use in otiier projects was one of the reasons tiie department adopted this approach for the 
National JTPA Study. However, tiie goals, setting, and basic structure of tiie JTPA study meant 
that implementation of its experimental research design was more complicated and difEcult than 
in past projects. As discussed below, tiie crucial distinction is that this study, for tiie first time, 
examines tiie entire, ongoing, voluntary employment and training system in participating sites. 

The characteristic of several past multi-site random assignment studies of employment and 
training programs are listed in Tabic 12. As shown in tiie top two rows of the table, most of the 
studies covered distinct program models or types of services that were offered to a narrowly 
deGned target group, rather than multiple services offered to a diverse group. In most cases, the 
studies involved fewer sites and individuals than were included in the National JTPA Study. 
Supported Work, tiie first project to use random assignment in a national, r.ulti-site lest of an 
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TABLE UZ 



OVERVIEW OF SELtCTED MAJOR RANDOH ASSIGNMENT 
STUOIES OF EMPLOYMENT ANO TRAINING PROGRAMS 



Chdmterfstlc 



Period of 
Project 

Pro9r«i» 

Intervention 

Studied 



j • Target ^roup 



Sites 



Experimental 
Sample 

Voluntary or 
Mandatory 

Run Inside or 
Outside Basic 
Service Oellvcry 
Systetn 

Small -Scale 
Pro9ra« or 
Large-scale ' 
System 



National 
Supported Work 
Demonstration 



1974-80 



Transitional 
ifork experience 



Long-term AFOC 
recipients* ex- 
addlcts, ex- 
offenders, young 
school dropouts 



10 local 
programs 



6,616 



Voluntary 



Primarily 
outside 



Small-scale 



WIN 

Laboratories 



ERLC 
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1978-82 



Job search, on- 
the-job 
training, and 
support services 



AFOC applicants 
and recipients 



3 WIN offices 
7,626 
Voluntary 
Inside 

Varying sizes 



AFOC Howewaker- 
Home Health Aide 
Demonstrations 



JOBSTART 



Texas Workers 

Adjustment 

Demonstration 



Demonstration of 
State Work/ 
Welfare 
Initiatives 



1983-86 



Training and 
subsidized 
eff^loyment as a 
home health ^Ide 



AFOC recipients 



7 states 
9,500 
Voluntary 
Inside 

Moderate-scele 



1984 -ongoing 



Basic education, 
occupational 
training, job 
search, and 
support services 



Young, 

economically 
disadvantaged 
school dropouts 
with poor 
reading skills 

13 local 
programs 

2,21.? 



Voluntary 
Inside 

Small-scale 



1984-86 



Job search and 

occupational 

training 



Dislocated 
workers 



Voluntary 
Inside 

Moderate-scale 



1982-88 



Primarily job 
search and work 
experience, but 
also on-the-job 
training 



AFOC and AFOC-UP 
applicants and 
recipients 



3 local programs 8 states 
2,300 



38,129 



Primarily 
mandatory 

Inside 



Major part of 
WIN system 



National 
JTPA Study 



1986-ongoing 



Varies by local 
program and 
needs of 
clients; study* 
Ing all services 
provided 

Economi cally 
disadvantaged 
adults and youth 



Up to 20 local 
programs 

Up to 30,000 



Voluntary 



Inside 



Entire locftl 
JTPA system 



TABU h2 (continued) 



Characteristic 



National 

Supported Work WIN 
Oewonitratlon Laboratories 



AFPC HoRtemaker* 
Home Health Aide 
Deoonstirationt 



J08START 



Texas Workers 

Adjustment 

Ocfwnstration 



Demonstration of 
State Work/ 
Welfare 
Initiatives 



National 
JTPA Study 



Studying 
Innovation or 
Existing Fro9ran 



Special Program 
Funding for 
Sites 



Innovation 



Yes 



Innovation 



ret 



Innovation 



res 



Generally 
innovation or 
enriched 
services 

Hodest special 
funding 



Existing program Innovation 



No 



Generally no 



Existing program 



No 



SOURCES! Hanpower Oetnonstration Research Corporation. Board of Directors 1980* le^iitan iqa?* d.h i . 

Gueron, 198;. • « ox uirecrors. lyw. Lefman. 1982; Bell et a],. 1986; Auspos et a1,, 1989; Bloom and Kulik. 1986; 
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employment and training program, was operated outside the usual "mainstream" service delivery 
system by specially created or designated agencies; the other studies invoked programs operated 
. '"ithin the mainstream system. In most past studies, the activities were small-scale, new, and 
innovative. Some were begun as part of a demonstration effort and would not have existed 
eroept for the research. In most cases, special program funding was available for participating 
sites, which were chosen primarily for their abiUty to implement the program and comply with 
study procedures. 

One way to summarize the challenges presented by the JITA study is to consider the 
continuum of these studies as they are ordered in Table 12. At one end is a pure demonstration. 
Supported WorL While it was a major undertaking and its success a breakthrough in research, 
the structure of the demonstration did offer advantages to the researchers. It tested the impacts 
of a carefully defined program model developed and funded as part of the research project and 
serving a particular group of participants. Further, it was operated by organizations whose mission 
was to operate the program as part of the demonstration. Control group members were excluded 
only from these special Supported Work programs. 

MDRC's studies of State Work/Welfare Initiatives were the closest to the National JIPA 
Study on this continuum. The Work/Welfare studies evaluated large-scale programs operated by 
the mainstream AFDC and WIN (Work Incentive) programs in many sites using a large research 
sample. In general, the sites received no special program support and often had to assume some 
of the costs of conducting the study. Prior to implementation of the JTPA study, the 
WorkAVelfare project represented the most ambitious use of random assignment field research 
to study the impacts of ongoing employment programs. However, these programs differed from 
JTPA because participation of eligible individuals was generally mandatory. The later discussion 
in this report will describe the importance of this difference. 

The National JTPA Study is at the opposite end of the continuum from Supported Work: 
It is an assessment of an already operating system offering a mix of services to a wide variety of 
participants with a research goal of changing existing program operations as little as possible and 
with no financial incentives for sites to participate. Memt>ers of the control group have to be 
excluded from all JTPA-funded programs in the SDAs in the study, rather than from a single 
service provider iu the comrnum'ty, which was the exclusion in many past studies. 

The JTPA study's unique characteristics complicated the design and implementation of a 
random assignmen'. field study. The great variety of local JTPA programs made developing a 
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single icseaich design a challenge. Study procedures had to be guperimpaed on SDA and. le^ 
provider admiarstrativc practices and performance assessment systems with as little disruption as 
. possible. The atent of the CKlusion of members of the control group, &om every JIPA-funded 
service provider in an SDA, heightened ethical concerns about the study and contributed to 
- making site recruitment a chaUenge. Site selccUon was made even more difficult by the absence 
of special program resomx«s for participating sites. 

IV. Aa,Qyerview of the .TTPA Rand om Assignment Field Stndv 

The RFP issued for the NaUonal JIPA Study outlined an ambiUous study, more complex 
than those previously undertaken. The study caUed for by the RFP moved random assignment 
research in many new directions; while no smgle advance was dramatic, together the departures 
added up to a project that went well past previous worL 

TTie department made a decision, in effect, to see how far random assigmnent Geld studies 
of employment and training programs could be advanced over past worL In some areas, this 
attempt was successful; for example, the study wiU yield internally valid estimates of both the 
overall program and categories of activiUes in the sample sites. Ho^'Dver, in other areas - for 
example, the effort to recruit a statisUcally representative sample of local programs - the research 
team did not succeed and the effort delayed the project and diverted resources from other tasks. 

As a result of the conscious choice by tht department to seek very ambiUous - and 
sometimes conflicting - objectives, the original RFP included features that were not part of the 
final research design implemented in participating sites. Table 13 summarizes key aspects of the 
original and final random assignment Geld study research design. As the table iUustrates. the 
research plan put in place can accomplish most of the goals of the project: The final research 
design includes a large sample of adults and youth, studies JITA in participating sites with few 
major changes, provides impact estimates for the entire JTPA program in these .Ues. and 
estimates impacts for several types of JIPA activities. The major change from the original design 
is the shift from sites in the study that are strictly representative of the JTPA system to sites that 
fllustrate the diversity of the JTPA system but are not randomly selected. Table 1.4 lists the sites 
in the study. 

The random assignment model developed for the study is shown in Figure 1.1 and is 
discussed in detafl in Chapter 4. SDA and/or service provider staff recruit applicants for the 
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TABLE 1,3 

SUMMARY OF RFP AND FINAL RESEARCH DESIGNS 



Feature 



Sites 



RFP Plan 



Up to 20, chosen to 

statistically 

represent the JTPA system 



Final Research Design 



16, chosen to illustrate 
the diversity of the JTPA 
system 



Sample 



Up to 30.000 adults and 
youth eligible for Title 
IIA 



20,606 adults and out-of- 
school youth eligible for 
Title IIA 



Evaluation of JTPA "As Is* 
with Little Change in the 
Program 



Yes 



Yes 



Evaluation of JTPA as a 
Whole 



Yes 



Yes 



Evaluation of Specific 
Treatments 



Focus was on specific 
activities, such as OJT, 
classroom occupational 
training, and job search 
assistance 



Focus on the types of 
combinations and sequences 
actually provided in JTPA, 
including categories of 
activities anchored on OJT 
and classroom occupational 
training 



Services for Which the 
Control Group Is Eligible 



Services in the cormiunity 
not funded by JTPA 



Services in the coninunity 
not funded by JTPA 
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TABLE lA 

SELECTED CHARACTERISTICS OF SDAs PAwriCIPATING IN THE KATIOHAL JTPA STUDY 



SDA 



Region 



Program 
Size* 



Largest 
City 



Final 
Sample 



Capital Area, MS (Jackson) South 



Concentrated Employment 
Program^ HT (Butte) 

Coosa Valley. GA (Rome) 

Corpus Christ!/ 
Nueces County, TX 

Crawford/Hancock/Harion/ 
Wyandot Counties, OH 

East Central Iowa 
(Cedar Rapids) 

Greater Omaha, HE 

Heartland, FL (Lakeland) 

Jersey City, NJ 

Larimer County, CO 
(Fort Collins) 

Hacon/De Witt Counties, 
IL (Decatur) 

Northeast, IN 
(Fort Wayne) 

Northwest, HN (Crooks ton 
and Thief River Falls) 

Oakland, CA 

Providence/Cranston, RI 
Springfield, MO 



West 
South 

South 

Midwest 

Midwest 
Midwest 
South 
Northeast 

West 

Midwest 

Midwest 

Midwest 
West 

Northeast 
Midwest 



Small 

Medium 
Medium 

Medium 

Medium 

Small 
Medium 
Large 
Medium 

Small 

Smal I 

Large 

Small 
Medium 
Medium 
Medium 



Jackson 

Butte 

Rome 

Corpus Christi 
Marion 

Cedar Rapids 
Omaha 
Lakel and 
Jersey City 

Fort Collins 

Decatur 

Fort Wayne 

Thief River Falls 
Oakland 
Providence 
Springfield 



1,478 

683 
1,840 

1,609 

1,154 

498 
1,362 

597 
1,686 

1.027 

471 

3,608 

560 
1.072 
1.759 
1.202 



NOTE: «SDAs were categorized within a region based on the number of Title IIA program 
teminees in program year 1984, the last year for which data were available when site 
selection began. 
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FIGURE 1.1 

RANDOM ASSIGNMENT MODEL FOR THE 
NATIONAL JTPA STUDY 



RscruMment 



Slgiblllty 
Dotermlnatlon 



Assessment 




^ Recom mended For 



Classroom 
Training 
Activity Category 




Services 



Control 
Group 



Other 
Services 
Activity Category 




program, detennine among them are eligible for JIPA services, and conduct their usual 
assessment of appHcants' skills, needs, and interests.22 Following the assessment, staff decide 
^which activities would be appropriate for the applicant and designate a treatment category based 
on this choice of activities. Ihey then call MDRC for random assignment Approximately two- 
thirds of all persons are randomly assigned to the experimental or treatment group, which is given 
access to the services designated by staff! The remaining one-third are members of the control 
group, who are not to be served within JTPA for 18 months. 

V. The OntHae of This Rep ort 

Chapter 2 summarizes key features of the JTPA program that affected the implementation 
of the study and presents the key concerns of SDAs about the project Chr^pters 3 and 4 discuss 
how the department and research team attempted to address these concerns. Specifically, Chapter 
3 discusses the response to the basic concerns of SDAs about the ethics, legaUty. pubUc relations, 
and costs and benefits of participating in the study. Addressing these was necessary but not 
suffident for a site to participate. Thus. Chapter 4 discusses the sites' other major operational 
concerns. It presents reseaYch questions and associated tradeo£& faced in dealing witii U:ese 
concerns so that a plan for the National JTPA Study could be developed and implemented. 
Chapter 5 ouUines the site selection process and compares the characteristi-cs of participating 
SDAs with sites that chose not to participate and with the JTPA system as a mholc It also 
briefly describes the study sample and discusses how the results from the SDAs in this study can 
contn-bute to an understanding of the JTPA system as a whole. Chapter 6 summarizes the lessons 
for future research coming out of tiie early implementation of the National JTPA Study. 
Appendix A provides further background on JTPA for readers unfanuliar with tiie program, while 
Appendix B contains brief profiles of the SDAs participating in the study. Appendix C presents 
comparisons of participating SDAs wiUi SDAs that decided not to partidpate in the study and 
with the national pool of SDAs from which study sites were recruiied. 

This report covers the implementation of the random assignment field study through 
September 1989, the end of random assi-joment in aU sites. Later reports. listed in Table 1.5. 



is done by SDA suff in most SDAs. However, in some, the recruiting and intake process is 
subcoDiraacd to the service providers who condua the uaining or operate the education or lob search 
assistance programs funded by the SDA j 



assistance programs funded by the SDA 
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TABLE 1.5 

FORTHCOMING REPORTS ON THE NATIONAL JTPA STUDY 



Date 

February 1991 
Deceniber 1991 
December 1991 
December 1992 

February 1991 
February 1992 



Topic 

Experimental Studies 

Baseline report on characteristics of the sample at 
random assignment 

Final implementation report on characteristics of 
participating SDAs and their programs 

Preliminary impact report based on approximately 18 
months of post-random assignment follow-up 

Final impact report based on approximately 30 months of 
post-random assignment follow-up 

Nonexperimental Studigs 

Preliminary reports on selection and participation and 
on nonexperimental methods 

Final reports on selection and participation and on 
nonexperimental methods 
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win present the baseline characteristics of the members of the research sample, a description of 
the programs under study in the participating SDAs, the early impact findings based on 
approximately 18 months of follow-up after random assignment, and the final impact analysis 
based on approximately 30 months of follow-up. In addiUon, separate reports will be issued 
ooalyzing the participant selection process in four of the SDAs in the study and newly developed 
noncjperimental methods intended to measure the average outcomes JTPA participants would 
cjqwrience without JIPA services. 
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CHAPTER2 
THE JTPA CONTEXT AND SDAs* CONCERNS 



The NaUonal JTPA Study was difficult to implement at the local level This chapter 
highlights particular features of the JTPA system, such as the decentralized operations, mulUpIe 
decision-makers, performance standards, and variations in the programs, and explains why they 
created challenges to conducting the study. The remainder of the chapter reviews SDAs* and 
states* perceptions about the study and the issues raised about the interaction between local 
programs and the random assignment methodology. Hie information presented in ibis chapter 
is based on the researchers' exchanges with SDAs and states during the site selection and 
recruitment process. It draws on telephone contacts with more than 200 SDAs and on site 
visits with approximately half of these. The purpose was to assess their viability as study sites 
and to determine their interest in and abiUty to participate. Actions taken to accommodate 
then- concerns are described in Chapters 3 and 4. 

L Key Features of the JTPA System 

Both the overall structure of the JIPA system and the specific requirements in Title HA 
influenced local and state ofGcials* reactions to the stucfy. Although JTPA is ahnost entu-ely 
federally funded, it provides states and local areas with the flexibility to select, within legislated 
parameters, the administrative and oversight structure of the program, the population to be 
served, and the types of services to be offered. A federally mandated - but state-administered 
- system of performance standards holds SDAs accountable for program outcomes (see 
Appendix A). 

The following features of the JTPA system for Title DA were espedally important in 
shaping state and local reactions to the study. Appendix A provides additional detail on 
JTPA*s funding structure, tiUes, expenditure requu-^ments and limitations, targeted populations, 
administrative and oversight structure, and performance standard system. 

• JTPA is decentralized, fostering a local orientation in SDAs. 



-23- 



ERIC 



The Job Training Partnership Act, Public Law 97-300, was passed in October 1982, at a 
time when the federal government was attempting to lessen its.invoh«ment in social programs 
and provide additional autiiorify to the states and locaMties. As a result, tiie Act establishes 
a decentralized decision-making and funding structure. States and SDAs make major decisions 
about program design and operations within Uie parameters established by tiie statute. 

Federal funds for programs operated under Tide HA of the Act are allotted as grants to 
tiie states, and states are required to allocate 78 percent of their HA allotment to tiie local 
SDAs. Tbt governor of each state is required to establish a State Job Training Coordinating 
Council, submit plans to tiie U.S. Secretary of Labor for tiie use of funds witiiin tiie state, and 
cany out otiier oversight functions. In addition to approving SDAs' plans for tfie 78 percent 
funds, tiie governor also plans and provides for tiie use of other TItie HA funds, including 
tiiose targeted to train older individuals (3 percent of tiie Title HA allotment to the state), 
cooperative agreements and coordination activities witii state education agencies (8 percent of 
flie HA allotment), incentive grants to SDAs (6 percent of tiie HA allotment), and state 
administrative and auditing activities (5 percent of tiie HA allotment). Hie governor is also 
responsible for designating SDAs, which are geographically deGned areas responsible for 
administering JIPA at tiie locc! level However, if requested, SDA status must be conferred 
on um-ts of local government or, generally, consortiums witii populations of 200,000 or more. 
The decentralized funding, oversight, and service delivery structure have enabled tiie system to 
focus on meeting local needs. It has also created variety in tiie subgroups targeted for services, 
the types of program components provided, tiie duration and sequence of activities, and the 
organizations tiiat deliver them. At tiie same time, because JTPA is almost entirely' federally 
funded,^ local areas do not have a direct financia' stake in ensuring tiie cost-effectiveness of 
the program. 

Hie decentralization of JTPA "distances" tiie local programs from tiie federal level. 
Many SDAs viewed the poHcy questions to be addressed by tiie study as relevant at the federal 
level but not particularly beneGdal to them individually. 



^uc.2n J T "^^P/""""- Sectons 202 (b)(1) and 123 stipulate that 80 percent of the 8 percent 
education set-aside mm be matched by the states and localities, but they may indude the dirca^s^o 
cmployincnt and training services they provide to JITA (2) Under Sca.on 30^ 'o qua rjiuS: 
III funds, sutes must match their federal allocation from public or pnvate non- edera Z^^^ 
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• JTPA is stnictured to involve saultipk dedsion-makers sf the local level 

Hie Act requires the establishment of a Private Industry Councfl (PIC) for each SDA 
and directs the PIC and the chief elected o£ficial(s) to share responsibility for decisions on the 
best use of funds within the local SDA. The Act also mandates that the PIC include 
representatives of economic development agencies, the pubUc employment service, individuals 
with handicaps, labor, education, community-based organizations, and businesses - with 
representatives of the private sector in the majority. The official decision structure is often 
augmented at the local level by the involvement of service providers, SDA staff, and social 
service agencies. 

Each of these groups, regardless of whether it had an actual vote, could influence local 
decisions such as whether to participate in the NaUonal JTPA Study. Each group weighed the 
implementation process for the study and the long-term benefits against its own and its 
constituents' interests. 

In discussions with local groups, some differences in orientation became clear:^ 

• PICs, with the majority of their members representing the private sector, 
were generally interested in the bottom-line assessment of performance that 
the study would provide. But they were also worried about the possibility 
of negative publicity from establishing a control group. They feared that 
their public image could be jeopardized if there were negative reactions from 
individuals and organizations in the community. Some were concerned about 
adverse reactions from employers who use JTPA as a referral source for 
trained or trainable applicants. PICs were also concerned about the study's 
potenUal effect on performance standards and costs for services. 

• Local elected ofllcials (LEOs), as representatives of a large and diverse 
constituency, questioned the legality of random assignment They too were 
especially interested in the potential effects of the control group on the 
program's image, the potential for negative publicity, and the political 
consequences of turning individuals away from services. However, they were 
also aware of the importance of having reliable information about program 
effectiveness for budgeting and other purposes. 

• Program administrators, while recognizing the need for the study, were 
concerned about how it would influence their staffs workload and the 
nature of their interactions with applicants. Program administrators also felt 



^is is a summary based on field contacts during the site recruitmg phase of the study. 
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responsible for ensuring the success and efifectiveness of the program and, 
as the .key staff charged with making recommendations and carrying out 
policy decisions, helped present the potential risks and benefits of the study- 
to the PIC and LEOs. The key role of staff is evidenced by the fact that 
a decision to participate in the study was never made without SDA 
managers' support In some cases, initially supportive PIQ and local elected 
officials were convinced not to participate by SDA staflt 

► Service providers under contract to the SDA were concerned that assigning 
individuals to a control group would change their lole and image as 
advocacy organizations. Many were operating under performance-based 
contracts, which make payments conditional on reaching specific goals. TTie 
control group alarmed them because if it negatively impacted on their ability 
to meet enroUment and performance goals, they could suffer financial 
penalties. As the group fartiiest removed from the federal level of JTPA 
m its daily program operations, and at the same time Uie one most 
financially pressured to perform, in many cases this group was also the 
hardest to convince of the benefits of the study. Service providers often 
viewed tiie study as a potential cost witiiout a sufficient of&etting benefit 
to themseh^es and their cUentele. Consequently. Uiey often sought to 
discourage participation in the study through direct or indirect representation 
on Uie PIC, or through their political influence in tiie community. 

State officials had misgivings about providing special consideration to SDAs 
serving as study sites in the event Uie control group resulted in lower 
performance or expenditures. TTie Act requires governors to allocate 
incentive funds (6 percent funds) among SDAs in equitable proportions 
based on Uie degree to which performance standards have been exceeded, 
but governors also have Uie flexibiUty to make special adjustments to 
performance standards. The need for "governors' adjustments" was viewed 
as unlikely to arise. However, because a random assignment field 
experiment of this magnitude had never been done in an SDA some SDAs 
were reluctant to participate without assurance that their participation would 
not negatively effect Uieir ability to qualify for incentive funds. The amount 
and nature of the adjustments that might be needed was unknown. States, 
in turn, were reluctant to make undefined commitments. They anticipated 
difficulty in establishing direct ties between changes in performance and 
participation m Uie study. Yet v/iUiout such evidence, it would be hard to 
get the other SDAs in the state to vie,w any adjustments as equitable. At 
the same time, since state assurances were important to SDAs, who placed 
a premium on incentive funds, state neutrality or inaction could also 
discourage participation. 

Community representatives were concerned about the ncjeds of people 
turned away and the potential strain on other service agencies to which 
controls might turn. Advocacy groups for specific subgroups of the 
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population were also concerned about the study's possible effect cn 
achieving targeted enrollment levels for their tsoastituencies. 

The decision to participate in this study required that each of these groups consent or, 
at a minimum, be willing to cooperate. The need for and benefits of tiie study had to be 
reviewed not just once, but many times. Typically, these groups would not aU be part of the 
formal approval process. However, because of the sensitive nLiure of this study, some SDA 
staff sought to involve groups that had not previously been involved in substantive pohcy 
decisions. In tiic participating sites, Uie upfront review and buy-in of the multiple groups, 
while time-cousuming, has been beneGcial during the implementation phase. Potential issues 
were discussed and solutions were developed in advance. Coupled with the sites' understanding 
of the purpose and procedures for Uie study, this enabled the study to proceed with a 
minimum of disruption. 

• Many SDAs had to exert considerable effort to nice! carollment and 
expenditure targets, particularly for some targeted groups. This raised 
serious concerns about establishing a control group. 

SDAs and states need to meet overall enrollment goals, goals for the enrollment of 
specific target groups, and expenditure plans. This section discusses why these three goals 
presented issues for the study. Section 141 of tiie Act specifies that the service delivery system 
is to provide employment and training opportunities to those '.vho can benefit from, and who 
are most in need of, such opportum'ties. The system is also expected to make efforts to 
provide "equitable services' t Tiong "substantial segments" of the eligible population. Except for 
explicit requirements to serve dropouts and recipients of Aid to FamiUes vwth Dependent 
Children (AEDC), these two tenris are not defined in the .Act or regulations. JTPA also 
requires that not less than 40 percent of the Title UA 78 percent funds is to be spent on 
eligible youth. 

In the early years of JTPA, this was often a problem for SDAs. To achieve the 40 
percent level of spending, they had tc recruit and enroll youths in greater numbers than they 
represented in the eligible population.^ Although the problem has diminished in recent years 



•^DAs may include services for in-school youth as part of their plan. 
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(only 10 percent of SDAs contacted cited it as a concern), SDAs had to be sure, that 
participating in the study would not jeopardize their meeting the requirement^ 

Many SDAs worried that diverting one-third of tiieir eUgible and appropriate appKcants 
to the control group would eitiier compound current problems with low enroUments and 
expenditures or create them. In recent years, especially with tiie decrease in tiie 
unemployment rate, many SDAs and their service providers have faced recruitment problems. 
The improvement in the labor market has meant that the more job-ready could obtain 
employment. Thvs, in most Icsal areas. JTPA has reportedly had to seek out otiier individuals 
who have fewer skills and weaker comiections to tiie labor market and who are typicaUy 
viewed as harder to serve. 

However, since JTPA funds are. on average, sufficient to provide services to only 5 to 
10 percent of the eligible population, tiie department and tiie researchers expected tiiat an 
increase in tiie level and quaHty of recruitment activities would produce tiie additional numbers 
needed to maintain enroUment levels, despite assigmnent of a portion of tiie appUcants to tiie 
conti-ol group. 

Lower-tiian-planned enroUment levels and difGcuIty retaining those who did enroU 
sometimes led to underexpenditure of tiie funds aUocated to tiie local leveL^ Consequently. 
SDAs were also concerned about how establishment of the control group would influence tiiei^ 
expenditures. 

In addition, some states have rules to recapture funds if expenditures faU below a pre- 
determined level, and otiiers tie the allocation of incentive funds to expenditure levels. Botii 
factors increase tiie importance of SDAs' meeting planned enrollment and expenditure levels. 

SDAs freqisently reported tiiat they needed to recruit up to four times more people than 
they had funds available to serve because of a large drop-off rate between the point of initial 
inquiry about JTPA and actual enrollment in tiie program. TTie enroUment and selection 
process often furtiier reduced Uie pool of eUgible applicants in many SDAs. 



.h, !j^^"!fn"^''°"^' ^ P*'""^ However, some sutcs and SDAs still fall shon of 

irLZ™enr..rr'- ^"^^^ °^ «^'"P'^'y y°"«h Programs, including one tha. a 10^^^ 
^ ffi^ aged 14 and 15. are described in Senior. 205 of the Act. 

•alloJn^ rS? . J ? 1'"^ ^""^^ employment and training program that preceded JTPA). 

v^^liT^^u^V^" "^^'^ ^ participant spent in tra.nmg 

programs. n,.s permitted the pnme sponsors and sutes (the seivicc delivery system under CETA) to civf 
clients a source of inMme while they participated. v-t.«; lo give 
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Some SDAs sought to stay below the performance standard for expenditures per client, 
which would mean that even more individuals could be r^rved with the available funding. In 
. such cases, even greater effort would be needed to meet enrollment and overall program 
expenditure goals. Efforts to keep per-person costs down and improve efBciency also affected 
an SDA's outreach activiUes, When such activiUes went beyond word-of-mouth and 
interagency referral agreements, it was usually to fill particular program openings. SDAs 
sought to increase demand only to the extent necessary to use the available supply of slots and 
funds. 

All of the above factors, coupled with the voluntary nature of JTPA, made individuals 
at the local level reluctant to explicitly turn applicants away. Creation of a control group thus 
posed serious issues for SDAs and service providers. 

• Performance standards and distribution of 6 percent incentive funds result 
in an emphasis on successful achievement of short-term outcome measures. 

JTPA has established an extensive system of performance standards, which awards 
incentive funds to SDAs that exceed their standards. Those that do not exceed their standards 
receive technical assistance. If they fail to meet key performance standards for two consecutive 
years, they may be reorganized. SDAs' performance is evaluated at the federal, state, and local 
levels, according to predetermined outcome-based standards.^ From the inception of JTPA 
through program year 1987 (ending in June 1988), the performance results for SDA programs 
for adults were judged by the following standards: the percentage of adults who obtained 
employment; the percentage of adults who were receiving welfare when they enrolled in JTPA 
and who found a job; the average wage at placement in a job; and the program cost per adult 
who obtained employment For youth, the standards included the percentage who obtained 
employment, the percentage who entered employment or attained other quantifiable measures 
of program success,*' and the program cost per youth positively terminated. Beginning in 



In addition to the performance sundards, other quantiutivc measures of performance are enforced 
to varying degrees. These include requirements for expenditure levels, equiiable services, and general 
compliance with provisions of the An and regulations. 

The youth positive termination rate included those individuals >vho atuined employment compeienaes 
recognized by local private industry councils, completed a certain level of schooling, enrolled m further non- 
Title IlA training, enlisted in the armed forces, returned to school full-time, or, for 14- and 15-ycar-olds. 
completed specified program objeaives. 
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program year 1988, an additional standard was added to measure the percent of pjMple 
employed 13 weeks after they left the program. Fbr each measure, the Department of 
. I^bor set national levels which - at state option - can be adjusted to reflect the characteristics 
of those served and the conditions in the '.jcal labor market 

Hiese measures and standards have resulted in an emphasis on successfiil acliievement 
of short-term outcomes, a reliance on performance-based contracts, and, in many cases, some 
initial screening and seivices prior to the point of enroUmenL Local areas often focus on 
generating high "success story" rates, which are generally interpreted as the highest placement 
rates at tiie lowest possible costs. Some state and local areas have independentiy defined 
targets, which moderate the emphasis on exceeding standards. However, tiie national average 
for expenditures per adult who enters employment and per youtii who positively terminates has 
coflsistentiy been at a level well below the national standard. 

While a few states have adopted tiie benchmarks established by tiie U.S. Department of 
Labor, nearly aU have adjusted tiie national benchmark based on local factors (using a model 
provided by tiie department). Some states have added fiirAer standards and a few have 
developed alternative performance standard systems. Hius, states vary in tiieir metiiods for 
allocating incentive fiinds. 

The importance of doing well on tiic performance standards is reinforced by tiie 
distribution of Utie HA 6 percent incentive funds based on tiie extent to which tiie SDA 
exceeds tiie standards. As explained previously, tiiese funds are available to the governor to 
provide incentive grants for programs exceeding performance standards, including incentives for 
serving hard-to-serve individuals. If tiie fuU amount is not needed for incentive grants, the 
remainder is to be used for techm'cal assistance to SDAs that do not qualify for the incentive 
grants. 

In recruiting sites, we found tiiat states and SDAs varied in the importance they attached 
to meeting and exceeding speciGc standards of pcrfbrmance.8 Incentive fiinds were also 
distributed in a number of ways to reward SDAs for exceeding individual perf-ormance 
standards. SDAs that received substantial awards and for which the awards were particularly 
important were less inclined to take risks tiiat might jeopardize performance outcomes. 



8 

Research done for ihc National Commission for Employment Pohcy found similar vanaiious Sec 
Natjonal Commission for Employment Policy, 1988. 
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However, there was fiEiirly consistent concern that some people assigned to the control group 
would be among those likely to succeed in the program and hence help the SDA meet 
performance benchmarks and standards. On the other hand, some states and SDAs 
encouraged the targeting of hard-to-serve subgroups and the development of special programs. 
These special recruitment targets also complicated implementation of the study at the local 
level 

An SDA*s reacUon to the study could be affected by how well the SDA was doing in 
meeting its goals, how important the receipt of incentive funds was to its overall program (in 
terms of proportion of dollars, flexibility of dollars, or prestige), and how much importance 
others at the state and local level attached to goal attainment 

The performance standards, coupled with JTPA^s rules limiting administrative 
expenditures, led to frequent use of performance-based contracts. Such contracts tie allocation 
of budgeted funds to the achievement of pre-speciGed performance benchmarks, such as the 
achievement of competencies or the placement of individuals in unsubsidized employment The 
existence of these contracts added to SDAs* and particularly to service providers* concerns 
about the diversion of randomly selected individuals away from JTPA services. 

In general, SDAs and the service providers they contract with viewed as a substantial risk 
the establishment of the control group and the consequent diversion of some of those 
applicants judged by them to be likely to succeed 

• JTPA does not follow a single program model. Services are individualized 
to meet local conditions and the needs of each SDA's customers (employers 
and Job seekers). 

Section 141 of the Act states that training is to be provided only for occupations in 
demand in the local area or in an area to which the individual is willing to relocate. SDAs 
view their ''customers'' as the economically disadvantaged individuals seeking training and jobs, 
the employers who will hire the job-ready client (the ^output" of the SDA). or both, with 
varying emphases. 

?lCs and local elected ofBcials a ;prove the job training plan for the SDA and thereby 
decide which employment and training services, from the array of available options allowed by 
JTPA will be provided. For the economically disadvantaged client, an employability 
development plan based on an assessment of the individual's needs and interests is iniiiall> 

.Sl- 



id 

ERLC 



54 



developed; the plan may be changed in response to changes in the person's needs and 
opportunities. The plan is generally individualized in terms of duration of services, number of 
steps, and sequence. But it is limited by the activities the SDA has planned to provide with 
the available funds. Some SDAs include options for the provision of basic education sidlls, and 
others emphasize shorter-term programs for the more job-ready, such as job search assistance 
and on-the-job training (OJT). Qassroom-based occupational skflls training can be provided 
through pre-odsting courses at local education institutions or through specially designed 
intensive programs that focus on the acquisition of a single skill or set of skills. The latter are 
sometimes designed in cooperation with specific employers to help them ffll particular vacancies 
and may be coupled with OJT. 

When the component(s) originaUy determined to be appropriate does net result in the 
development of marketable skills or. after further assessment, is not acceptable to the peison 
receiving the services, the plan may be changed. In some SDAs. assessment is incremental, 
with each phase producing decisions only about the immediate next step. Job search assistance 
is often used as a starting point and serves as both an extended asse^ment and a labor market 
screen. 

For the employer, individualized services also include the referral of "best" candidates for 
jobs or OJT positions, quick processing of any employer-referred candidates, and simple 
procedures with a minimum of red tape. Some SDAs have established themsebes as a 
persomiel office for local firms, either independently or in coordination with the public 
employment service. 

TTierefore. actions that inhibit an SDA's flexibility in meeting the needs of these 
customers raises concerns. The study requires, for example, deciding up-front what category 
of services an individual is to receive and not changing the decision as the person mova 
through the program. It also requires that candidates for training or OJTs. including those 
referred by employers, be randomly assigned. 

General C oncerns About the Study at the Local Level 

In discussions with SDAs. ti:? research team had to address the concerns of the many 
actors at the local level. These concerns pertained primarily to general ethical and public 
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relations issues and specific operational and perfonnance-related questions about the study's 
methodolo^. 

Agreement to participate in the study was actually a two-step process. Fust, the SDA 
and service prov;der staff and PIC members had to understand and generally approve of the 
study. For purposes of presentation, state and SDA quesUons about the purpose of the study 
and the methodology it employed are identified in this report as "general" concerns. An SDA 
had to be satisfied about these matters before it would even consider participating in the 
study. Partidpating in a random assignment field study was different from anything SDAs had 
previously been requested to do. Consequentiy, the discussions were lengthy and detaUed, 
Second, once the reason for the study was accepted, SDAs had to consider what it would mean 
to implement the study whfle fulfilling their ongoing responsibilities. The questions pertaining 
to actual implementation of tiie study design and the rules sites would have to foUow are 
labeled "operational" concerns. 

The remainder of this chapter explores the risks or difficulties the SDAs and state staff 
envisioned if they were to join the study. The discussion begins witii die general issues, those 
pertaining to the use of an experimental methodology. It concludes wiUi the operational 
issues tiiat had to be addressed in order to implement the study. 

Table 2.1 summarizes the major issues raised by 228 SDAs witii which the researchers 
had some formal contact about possible participation in the study.^ The concerns discussed in 
the remainder of this chapter are derived from this table. It is important to note that the 
listing includes those issues that were voiced by sites; it is not the result of a response to a 
survey. 

A. The Need fo r the Study and the Value of Parficip af 
1. Resistance to additional studies of JTPA. 

Most SDAs were not eager for an additional study. People at the local level often 
believed that the performance standard system and monitoring and auditing activities of the 
state and federal government produced more than sufiicient information. A number of SDAs 
indicated that they had recently been the subject of federal studies; since participation in this 



*nie process of site rccniitmcm and ihc nature of these coniacu arc desaibed m Chapter 5. 
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TABLE 2.1 



PERCEHT OF SDAs CITIKS SPECIFIC 
CONCERNS ABOUT THE STUDY 



Concern 


Percent of SDAs Citing, 
the Concern 


Ethical and Public Relations loplications of: 
Randoa Assignment in Social Prograes 




0«iial of Services to Controls 


54.4 


Potential Negative Effect of Creation of 
a Control Group on Achievaeent of Client 
Recruitisent Goals 


47.8 


Potential Negative Icipact on Perfomsance 
Standards 




laplenentation of the Study Hhen Service 
Providers Do Intake 




Objections of Service Providers to the Study 


17.5 


Potential staff Administrative Burd:n 


16.2 


Possible Lack of Support by Elected Officials 


19.8 


Legality of Random Assignment and Possible 
ari evances 


14.5 


Procedures for Providing Controls with Referrals 
to Other Services 


14.0 


Special Recruitiaent Problems for Out-of-School 
Youth 


10.5 


Sample Size 


228 



SOURCE: Based on responses »,f 228 SDAs contacted about possible participatun in 
the National JTPA Study. f v in 

NOTES: Concerns noted by fewer than 5 percent of SDAs are not listed. 

Percents siay add to eore than 100.0 because SDAs could raise more than 
one concern. 
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study was voluntaiy, they chose to avoid additional intrusions. Some individuals felt that JTPA 
was already under greater scrutiny than other federally funded programs because of its 
perfonnance standards; they beh'eved that further investigation was unwarranted. 

2. Confusion over impacts and outcomes. 

The JTPA random assignment study was being introduced into an existing, weU- 
established system, in which performance standards have been heavily reKed on as indicators 
of program effectiveness. At the local level, SDA staff often encouraged their Pia and 
elected ofHcials to judge program success by outcomes on performance standards. Hie 
allocauon of incentive funds from the states to the local level remforced this emphasis. Except 
in rare cases, people at the local level were not familiar with the difference betv/e«n impacts 
and outcomes and the past unsuccessful efforts to measure impacts without random, assigament 
Therefore, they iniUally had difficulty grasping the necessity '.r a random assigrjnent design. 
In general, poKcymakers, administrators, program operators, and line staff were accustomed to 
gauging program effectiveness by their results on the performance standards. 

3. Reluctance lo participate as one of a smaU number of sites in a 
study designed to benefit the Siystem as a whole. 

The Nauonal JTPA Study was designed primarily to provide federal poKcymakers with 
information on JTPA's effectiveness. SDAs often did not feel a responsibiUty to participate, 
especiaUy since participation was not mandated and only 15 to 20 sites were needed out of a 
pool of 504 that feU within the broad research parameters for size and location. Many of 
those contacted felt that their individual decisions not to join would be inconsequential. Some 
SDAs said that they could get the results by comparing theu" SDA with others that did 
participate, without the burden of the research. 

As noted earher, JTPA's decentralized funding and decision-making structure contributed 
to the reluctance to be part of a national study. Hist, since JTPA is almost entirely federally 
funded, the financial rewards to the state and locality from being cost-effective are less direct 
than they are in other systems. For example, funding for welfare cash grants and for 
employment programs includes, in most cases, state and/or local tax dollars, in these programs, 
state and local governments therefore have a direct financial incentive to understand the 
factors that influence the return on their investment in training and the size of this return. 
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STPA dccfcion.make« at the local levei did, however, understand the importance of 
detemiimng the cost^&ctiveness of JTPA, for other reasons. Tte pubKc/i,rivate partnership 
established by the A« was mtended to improve the responsiveness of the ^tcm to labor 
demand. But it ^ also a ^ to improve local accountabiUty as to the efficiency and cost- 
effectiveness of service deliveiy. I^k^I elected officials remain invoked in JTPA through 
appointing members to the PIC They also serve as partners to the PIC in approving the local 
plan and the selection of the organizations to be responsible for planning and administering 
the program. At the local level, the actual roles assumed by the decision-making bodies varied 
^dely. but most groups focused im'tially on the potential influence of establishing a control 
group on their programs and on relationships in the commum'ty. 

Second, because of JIPA's structure, local program operations are far removed from the 
federal level Over JIPA's seven years of existence. SDAs have developed their o^ 
commum-ty role and administrative and evaluation systems. Tl^us. SDAs were less inclined 
than agencies directly linked to the federal goverrmient would have been to feel responsible 
for partiapating in a study whose primary purpose was to benefit the system as a whole. 

Tlurd. the autonomy for state and local areas represented in the Act, coupled with the 
federal government's effort to pass greater authority to state and local govermnents. meant that 
once the Departmem of Ubor dedded to make partidpauon in the study voluntary.^" it was 
not able to exert much influence over a site's dedsion about joining the study. 

Fmally. while recognizing the need for the study nationally, some SDAs were concerned 
about the responsibiUty they would be assuming for providing information that could influence 
the future direction of the whole system. Some SDAs wondered if the results would be 
framed and timed by a poUti'cal agenda, and used in a way that could somehow be detiimental 
to tiie system. They were also worried about the local consequences if the story were to 
reflect negatively on aspects of their program or if they compared unfavorably with other 
SDAs 2n the study. 



of .V *° "^"^'^ partidpation was also supported by the researchers Th^ «,mn.^r, 
of the research design made the cooperation and commiunem of 1,^1 arc^^ntSi IJd 
paruapauon «uld have made U,e stud^far more difficult to c^rr^ ou^ ' '"^ '^'""''"^ 
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B. The Ethfcs of Random Assteninent 

Early in the discussion with virtually eveiy site, there arose quesUons about the ethics 
and public relations eEfects of randomly assigning volunteers to a control group in an under- 
enrolled system. As indicated in Table 2.1, 62 percent of the sites contacted about the study 
directly expressed concerns about the ethics and public relations aspects of random assignment 
as a research method. Some raised the issue spontaneoujly; others asked about it after 
researchers pointed out the typical quesUons that arise when a random assignment design is 
used. 

1. Unfamiliarity with the design of experimental research. 

As noted above, individuals at the local level often questioned the need for a rigorous 
study. In particular, they could not accept the need to establish a control group until they 
understood that the knowledge to be derived from this study was not currently available, and 
that other methods for obtaining information on imp'^^^ had been considered and rejected by 
evaluation experts as not sufBciently reliable. 

The very idea of unplementing an experimental design within a social service program 
was foreign to them, and there were often strong negative "gut" reactions to the concept of 
random assignment Approximately 70 percent expressed concern about the use of random 
assignment to determine who would be enrolled or the dem'al of services to controls. To some 
people, an "experiment" evoked images of laboratory experiments, which they objected to on 
moral grounds. Others supported the experimental method but did not deem it appropriate 
for a study of an employment and training program. Others questioned whether other types 
of comparison groups would not be sufGcient, especially when weighed against their perceptions 
of the intrusiveness of this design. 

2. Denial of service in a voluntaiy system In which much effort is 
sometimes needed to reach enrollment goals. 

Of the SDAs contacted about participation in the study, 54 percent indicated thai denial 
of services was a concern. Many people questioned the ethics of assigning volunteers to a 
control group. Often, they described their program as a virtual entitlement for all eiigibles 
who applied. They fell that by the very act of applying, applicants were demonstrating a 
special laeed or worthiness. They objected to making participation in the siud> a condition of 
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program cntiy by requiring an appUcant to sign the agreement to participate form early in the 
process of eligibib'ty determination. 

Often, individuals at the local level viewed Uieir program as serving eveiyone who wanted 
or needed services, even though funding limitations generally result in SDA services being 
available to no more than 5 to 10 percent of tiie eligible population. TTiey held tiiis view 
because recruitment was a petsistem problem. piCs were often surprised to learn that tiiere 
were many more individuals eligible than resources wtjuld enable them to serve. However, 
random assigmnent still meant tiiat Uie dem'al of service became active (actually saying no) for 
tiie period of the research raUier than taking tiie indirect forms of Umiting outreach efforts and 
intentionally or unintentionally establishing pre-enrollment steps to screen for "motivation" and 
"appropriateness." 

3. Treatment of Ibe control groap. 

Additional concerns were raised regarding tiie treatinent of contiols. Some SDAs felt 
tiiat controls should be allowed to receive some type of JTPA services, should be compensated 
for their exclusion from services, or >*ould be actively referred to otiier community services. 
Among tiie SDAs contacted. 14 percent felt constrained by not being able to actively refer 
controls. Otiiers expressed concern about tiic lengtii of time controls were to hct excluded - 
30 months. 

C Legal Issues and Potential Grievances 

SDAs were also concerned tiiat tiieir participation might result in grievances and legal 
chaUenges arising from tiie statute or civfl rights laws. As noted above, many individuals at 
the local level viewed JTPA as an entitiemcnt program, which it is not TTiere was also 
concern that random assignment was somehow discriminatory. Issues regarding the rights of 
the researchers to government records on individuals and procedures to ensure confidentiality 
were also raised. 

D. Public and Community Relations 

People were also concerned that, given the lengthy educational process it took for them 
to accept tiie study, the larger community might weU not understand or accept it In some 
SDAs. tiic possibility of unfavorable media coverage was cited. The community's reaction to 



-3S. 



ERIC 



61 



random assignment, or even to the recommendation to participate in the stu(fy, might diminish 
the program's broader appeal PIC members, conscious of their bottom-line orientation, did 
not want it to look as if they did not care about people. The possibility that new Leal 
officials might be elected who wre unsympathetic to the study, and other local government 
issues, were cited as a concern 16 percent of the time. 

JTPA's programs and services were part of the broader community. SDAs were 
concerned that other agencies, with whom they had long-standing relationships, might stop 
making referrals to them as they began assigning individuals to the control group. There was 
also some concern about the burden on oxher agencies if controls turned to them for services. 

SDAs often tried to fill gaps in the local service network. Thus, the local program was 
embedded in a network of services, fulfilling a function for social services and for employers. 
SDAs did not want to disrupt this network by establishing a control group. 

Consequences of Not R eaching Performance and Contract Coals 

SDAs were also concerned that establishing a control group could negatively impact on 
their performance measures, hurt their reputations, and lesson their likelihood of qualifying for 
state incentive funds. Some SDAs were concerned about theu" stature in the community if the 
impacts were not statistically significant 

If recruitment was insuffident to fill available slots, that would mean the SDA was not 
using funds available to serve disadvantaged people, an ethically troubling situation, especially 
if they were not allowed to cany forward funds and spend them later. Politically, it could 
produce the charge that the program was over-funded. 

in. State and Local Operational Issues 

In addition to raising questions about methodology, SDAs had to consider how the study 
would work within their own system. The major issues were the following. 

A. Disruption of Intake and Enrollment Procedures 

1. Influence on recruitment practices. 

SDAs were concerned that expanded recruitment efibrts to produce the numbers 
needed for the study might generate applicants who were less needy, less motivated, and 
therefore harder to serve. 
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2. Additional feoat-efid papsm>rl£ and Its effect ©n applicants. 

Early in the emomnem process, a baseline data c»llecUon form and an informed 
consent form had to be completed for each applicant EUgibiUty determination was already 
a cumbersome process, and the addition of new forms - the background information form 
alone was four pages long - was viewed as discouraging appUcants &om following through with 
the process. Since eligibiUiy determination often took a number of return visits, the decision 
to position random assignment after eligibiHty had been determined was raised as a concern 
IP 11 percent of the SDAs. 

^ -Ponseqtienres of Earlv A..t,>nn,.n« to ArtMfy n,fPon^w 

Ihe study metiiodology required individuals to be assigned to treatment groups prior to 
random assignment (but after an initial assessment). TTus was viewed as interfering ^'ih SDAs' 
normal, ongoing assessment process, limiting their abiliiy to change service recommendations 
and jeopardizing achievement of their performance standards. 

The assessment and reassessment process, coupled wiUi Uie array of choices available 
comphcated the assignment of individuals to treatment streams for the study. Tbis was the 
case even though, in practice, the im-U'al employment plan is seldom changed in any major way 
for most participants. Short-term services intended to lead to placement in .n unsubsidized 
job. such as job search assistance, and skills training programs turned out to be the most 
frequently emphasized activities. However, even within this pattern there was great variation 
in the tjpes of programs offered, and SDAs were reluctant to give up tiie flexibiUty to switch 
a person from OJT to skills training or vice versa if the original service plan proved 
unworkable. There was no one model of assessment or services, which made it difficult to 
impose Uie random assigmnent metiiodology and to isolate Uie components to be studied. 

In addition, multiple agencies were sometimes involved in the process of determimng 
cligibiUty and completing assessments to determine the appropriateness of services under JTPA. 
In SDAs where services were decentralized, operational problems were cited as a concern 21 
percent of the time, compared to only 5 percent of tiie time in centralized SDAs. 

In SDAs where services to employers are emphasized - including referrals, assessment 
assu^tance. customized training programs, and OJT slots - the process caused additional 
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concerns. The establishment of the control group was viewed as reducing the available s.uppfy 
of appropriate applicants to local businesses. 

C Staff Bnrden and Workload 

Recruitment, paperwork, explaining the study to applicants, and intervention if grievances 
arose were all viewed as potsctially time-consuming activities, which would divert staff 
resources fiom other worL Hie extra effort required to complete the research forms, make 
random assignment phone calls, and explain the study was a significant concern in 16 percent 
of the SDAs. There was also concern that disgruntled controls would approach elected 
ofScials and create an image of unresponsiveness to which SDAs* staff would need to respond, 

SDAs also had to weigh participation against other new ventures, including major 
organizational or programmatic changes. The lengthy time commitment was also a cause for 
concern. 

Potential performance problems could not be isolated to only one program year. The 
ori|,-inally proposed SO-monUi exclusion period for all controls (subsequentiy shortened to 18 
months) was viewed as both an ethical and an operational issue because SDAs would be 
unable to recontact tiiis group in the foreseeable future to help build enrollments. 

D. Reimburse ment for the Costs of Implemenf inp fhp Study 

The VS. Department of Labor authorized compensation to the sites for then- efforts in 
implementing the study. However, SDAs often questioned whether the funding for 
participation in the study would compensate for Uieu" costs and efforts. They feared that the 
study might divert funds from other services. Further, as staff efforts were diverted to 
implementing the study, staff would spend less time on sen/ices that could enhance 
performance. Participation could also count against them as costs per adult job placement and 
youth positive termination were calculated. 

E. Influence of the Control Group on Performance 

As already discussed, assignment of individuals to the control group could exacerbate 
recruitment problems, cause underenrollment, and result in the underexpenditure of funds. 
Recruitment was cited as a concern by 48 percent of the SDAs. Additional recruitment effons 
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would absorb staff time and might not succeed. Further, if the cfiforts led to enrolling less job- 
ready dients, the 6 percent incentive grant awards might be threatened. 

Commum'fy-bascd organizations serving as contractors to SDAs and rural SDAs were 
particularly concerned about the effect of denying service on their community image and ability 
to recruit They feared that word would quickly get around that some appUcants were being 
turned away. In small communities, there might be the awkwardness of encountering people 
who had been denied services or seeing them at community functions. 

F. Influence on SetMre Providers* miitv To Meet Performance Benr!imHrV« 

Objection of service providers to the study was cited as a concern 17 percent of tiie time, 
•nie regulations provide an option for Gxed-um't price (performance-based) contracting, an 
approach frequenUy used by SDAs in their subcontracts for services not provided by in-house 
staft niese contracts aUow 100 percent of the costs incurred in providing training and services 
to JTPA participants and overseeing tiie program to be charged to training. They have served 
as a vehicle for covering additional administrative costs." More importanUy for the purposes 
of the study, fixed-unit price contracts provide Uie SDA with a method of conti-acting tiiat 
ensures tiicy only make payments when goals are achieved. Some comractors mistakenly 
beUeved tiiat tiicy would have to enroU individuals whom they oUierwise would have excluded. 
OUiers felt tiiat to meet enroUment benchmarks tiiey might have to be generally less selective. 
Etiier way, many contractors worried about not being able to achieve tiie benchmarks 
necessary to recover the full amount of tiieir contract. 

The involvement of tiie private sector and the introduction of performance standards 
seem to have driven tiie system toward negotiating lowK»st contracts. TTiese have enabled 
SDAs to serve more people witii tiie same amount of funds, sometimes exacerbating recruit- 
ment problems. They have also intensified the pressure on contractors to meet performance 
goals. Thus, contractors take very seriously any threat to their enrolling sufficient numbers of 
people who are Ifkely to succeed. Interestingly, some PIQ tiiought the study would force 
vendors to be less selective, which they saw as a positive change. 



"Legislation is currently under consideration in Congress that would increase the adminLc.rative 
category from 15 percent to 20 percent 

The Dcparunent of Ubor has recently issued new guidelines on performance-based contracu which 
bave ughteaed what can be charged as a training cost Other new proposals would further rcstna their use. 
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G. Perfonnance Stnintards and 6 Percent Incentive TmtAs 

The influence of a control group on the achievement of performance standards and 
eacpendit'ire benchmarks was cited as a coLcem 25 p^^rcent of the time. In some SDAs, 
incentive grants were a significant portion of the budget; in others, they provided additional 
flexibility. State assurances were needed to overcome SDA concerns, but states were 
concerned about setting a precedent for governors' adjustments on performance standards and 
also worried about negative reactions from SDAs not in the study. 

States showed vaiying degrees of willingness to support the National JTPA Study, a 
federal initiative. Some questioned whether regional Department of Labor employees who 
audited them two or three years later would be supportive. 

In summary, whfle SDAs generally recognized the benefits of the study at the national 
level, there were many issues that needed to be addressed at the local level. The next chapter 
reviews how the general concerns were addressed, sometimes through changes m procedures 
or modifications of the design. 
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CHAPTERS 

ADDRESSIN G THE GENERAL CONCERNS OF THE SDAs 



To reach an agreement with an SDA to parUcipate in the study, MDRC staff had to deal 
with the many concerns described in Chapter 2 This chapter discusses hov; the fundamental 
quesUons about random assignment and the study were addressed. However, they were not 
always the first matters raised, nor were they necessarily the reasons an SDA decided against 
participating. The process of resolving the "operaUonal" issues - those relevant to making the 
design operationally feasible - is discussed in Chapter 4. 

!• The Ethics of Random Assignment 

As described in Chapter 2, ethical concerns centered on questions about the need for 
another study of the system, the use of an experimental methodology that would exclude some 
people from JTPA services, and the appropriateness of this exclusion in the face of SDAs' 
difGculties achieving desired enrollment levels. The length of time controls would be excluded 
was also at issue. 

A. Explaining the Need for the Study 

In long and detailed discussions, MDRC staff explaineu the need for the study and how 
the information it would provide could help SDAs improve their programs. They pointed out, 
for example, 'hat no existing study of JTPA attempted to estimate program impacts as opposed 
to program outcomes. (The distinction between outcomes and impacts was further clariGed.) 
MDRC staff also reviewed the history of past nonexperiraental research, reported in Chapter 
1, to illustrate the need for reliable random assignment research on program impacts. 

B. Addressing Ethical Concerns 

MDRC staff typically brought these matters up early in their discussions with SDAs, since 
virtually everyone was worried aboi* them. Staff stressed the need for rigorous research to 
improve the knowledge base of the system so that SDAs can build on strengths and correct 
weaknesses. Staff also clarified key points: 

i. JTPA is a slot-limited program: It is not an entitlement for all who meet 
the eligibility criteria, as is the case with some other state and federally 
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funded programs. Random assignment is usually at least as feir and 
equitable a ^ to allocate limited program slots as otber methods that are 
used, such as first^me, first-served or counsclore* and service providers' 
assessments of those most likely to succeed. SDAstaEf acknowledged that 
thqr did routmely choose not to serve some eUgible appUcants and 
understood that random assignrj ?nt only made the de' 'al of service explicit 
atid of longer duration. -fuwi 

2. Random assignment (coupled with effective assistance in cUent recruiting) 
should lead to a change in the way the SDA selected people to be servi 
but not m the number to be served. 

3. Basting recruituig and intake techniques did not assure that those applyine 
for program slots were any more needy or worthy than those who would 
t« iT^'l!^ ^ the expanded outreach that partidpating SDAs might need 
nf ^ ^'J^"? ^^^B random assigmnent In fact, the pool 
Of potentiaUy eligible appUcants was Ukely to include people with a wide 
range of skills levels and motivation, including some who might benefit 

Sml ^""^ ^"^ ^ *° 

^t^•^^f ^'^^"'^'^ umiecessaiy steps that screened 

out mdividuals who were appropriate for services. 

The provisions for informing program appUcants of the study's purpose and procedures 
reassured SDA staff that appUcants would understand the ground rules for entry into JITA 
and worJd be treated fairly and sensitively. The ".greement to participate" form, given to 
people early in the appUcation process, was develop<Ki and refined with the assistance of some 
sites. Its purpose was to sujiply and clearly cxplaii. the aims, process, and requirements of the 
study. This form was originally translated into Spanish. Later, it was translated into nine other 
languages when an SDA needing this material joined the study. 

Staff training and £ procedures manual containing a sample script, talking points, and 
frequemly asked questions also faciUtated the implementation of random asslgnmem during the 
intake process, as did a videotaped explanaUon of the study, developed with the Department 
of Labor and adapted in some SDAs to include local scenes and people. The videotape 
assured a consistent and complete explanation of the study, at least until staff became 
comfortable explaining it themselves. 

Hnally. the department authorized the research team to exclude particularly needy groups 
of applicants from the study if they did not constitute a large portion of an SDA's program. 
Examples of groups tlnal were allowed to bypass the random assignment process and directly 
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enter the program in some SDAs include homeless in(iividuals, pereons referred by courts to 
receive services as a condition of probation, and indKiduals with severe handicaps. In addition, 
the department allowed each SDA discreUonaiy exclusions &om random assignment equal to 
approximately 1 percent of the SDA's target number of random assignments.^ 

^ The Natnre and Dnrati nn of the Escclnsion of the Coatrol Group 

SDAs and service providers wanted to minimize the withholding of JTPA services from 
people assigned to the control group. IniUally, the research plan would have required SDAs 
to inform those assigned to the control group that they could not be served within JTPA for 
30 months foUowing random assigmnenL Im'Ually. too. there were no formal procedures 
penm'tting staff to suggest other service options to this group. Fifty-four percent of the SDAs 
contacted expressed serious reservaUons about denying aU JTPA services to controls; 14 
percent specificaUy objected to being prohibited from actively referring them to other agencies. 

In response, the research plan was modified in two ways. Fust, even though foUow-up 
was to continue for 30 months after random assignment, the exclusion period was reduced from 
30 to 18 months, after which time SDAs could serve controls // they returned to the program 
on their own im'tiative. A review of the literature on programs such as JTPA and discussions 
with SDAs and service providers indicated that most appUcants contact a program during an 
episode of unemployment or when specific events in their Uves lead them to seek training. 
This implies that most would not reapply to the program after 18 months if the exclusion were 
lifted at that time; thus, the service difference between the treatment group and the controls 
would, in reality, continue for the entire 30-month follow-up period. Information to be 
coUected on services received by controls - through reviews of JTPA program participation 
records and follow-up surveys - wiU reveal whether the shortened exclusion period did result 
in significant levels of JTPA service to controls after 18 months. TTie findings will be 
presented in later reports on this project 

Second, SDAs were allowed to provide all controls with a list of other, non-JTPA 
programs operating in the community but were not permitted to arrange individual referrals. 
MDRC and SDA stafi" in each site agreed on the list and the wording of the accompanying 



'Agreements to participate were negotiated with each study site and defined the sample size studv 
procedures, financial compensation to the sites, and other obligations of both the sites and the research 
team. 
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memo or letter, l^picalfy, the list induded the state empk^ment se^ jbe only 
alternative in rural areas). local community colleges and technical institutes, and community 
agencies providing social services and housing assistance. The level of receipt of non.JTPA 
services by controls (anu those in the treatment group as wD) wiD also be measured in follow- 
up siuveys,^ 

Other ideas were suggested by some SDAs but were rejected. One recommendation was 
to provide some type of payment to controls to compensate them for the time and money (to 
cover transportation to th. intake office and costs of ch3d care) they spem in applying for 
services and to partially make up for their not receiving program services. However, a small 
payment would have undervalued the service provided by the SDA, while a large paymem 
maght have biased the results of the control group and was beyond the budget for the research 
study. A second possibffity was to provide some short-term, ^intensity- JTPA services, such 
as job search assistance, to all controls. This was ruled out for three reasons: The research 
design required a control group that did not receive JTPA services; nc^t aD SDAs already 
offered this type of service, and developing it would have diverted funds needed for other 
purposes; and short-term job search might be the only JIPA activity some individuals in the 
treatment group would receive, so providing it to controls would erase the difference between 
some members of the control and ejqxjrimental groups. 

n. Legal and Pnblfc Relaffnns Tcchpc 

SDAs were provided with a legal opimon by the SoUdtor General's Office of the U.S. 
Department of Labor that random assigmnent was legal under JTPA. Further, MDRC st^ff 
were able to report that grievances and legal challenges had not been an issue in past 
evaluations this team conducted using random assigmnent - a track record made possible with 
the strong cooperation of study sites, which foUo^^•ed the research procedures developed to 
ensure legality, fairness, and sensitivity. 

Whether to actively seek press coverF . for the study was a difficult question. SDAs 
wanted to announce their participation in the national study, and some felt that providing 
clear explanations of the control group early in the process would prevem possible 
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misinformation in the future. MDRC prepared it press package including background on the 
Etudy and a press release, which SDAs could adapt for the:r own community. Most SDAs 
did not use it, deciding instead to e3g}lain the stud|y to the general public only if inquiries were 
made. In several 6DAs, MDRC staff were brought in to assist with briefings of local elected 
ofiSciaiS and social service agencies not directly involved with the local JTPA program. MDRC 
staff were also available to brief the press. 

in. Recruitment and Enrollment Issues 

Hiese concerns were addressed in several ways. First, the original random assignment 
ratio of one treatment group member for each control group member was changed to two 
persons in the treatment group for each control, allowing SDAs to enroll a larger pioportion 
of appUcants. This and other changes in the random assignment raUo are discussed in more 
detaD in Chapter 4. 

Recruitment concerns were also addressed through technical assistance on recruitment 
and retention techm'ques, using specialists who had previously provided similar training within 
the JTPA system. Among the lessons offered to SDAs and service providers were:^ 

1. View outreach snd recruitment as a normal part of program operations: 
Many staff implicitly assume that motivated, needy indhiduals will 
automatically seek out their services. They fear that recruitment efforts 
may, in fact, draw into the program less motivated, harder-to-serve 
appUcants. In many communities, those people who are in contact with a 
sodal service agency such as welfare or the state employment service may 
know quite a lot about services such as those offered by JTPA, but the 
general population may be uninformed. Enhanced recruitment efforts can 
be used to draw in tb's large and diverse population. 

2. Tai^t messages to reach subgroups in the eligible population: SDAs 
should identify key groups that are hard for them to recruit and develop 
customized messages to reach them. 

3. Sell the program^s benefits rather than its features: People want to know 
what thq^ will get out of the prograir. - how it will serve their needs and 
goals. Information that just describes the program's organization and 
services is far less effective. 




^Thc assistance included group training sessions b> Cygnet Associates, preparaiion of two lengihv 'how. 
tc* manuals by C^gnci, and on-site visits to manj of the stud> SDAs. Sec Kcll>, 1987, and Elsman, iybT. 
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4. Usse ll» rsoBltsaeBt effort optiiaany: By planning their cwitta/ recruit- 
ment and cnrollnient campaign in advance, agendes can time their ou 
e&m to Wt people vs^en thqr are "in a buying mood" and avoid the need 
for frantic efforts at the end of the program year in late June, a difBcult 
time to iccruiL Interest and participation in employment and training 
programs typically peak in the spring, fall, and period following the year- 
end holidays. 

Oient retenUon is just as important as recruitment. In many SDAs, fewer than one of 
evciy four persons making an inquiiy about program services is ever enroUed. Ihe others 
are prime candidates for SDAs' services, having already expressed an interest. Ihe technical 
assistance highUghted these lessons for client retention: 



1. 



Adopt the viewpoint of potential applicants: Think of appUcants as the 
agency s customers, who can go elsewhere if they do not like the sendee 
they receive. AppUcants can lose interest if they are forced to vrait for an 
extended period, are treated impolitely, or are served in dingy or depressing 
surroundings. Fusi impressions can be important: RecepUonists, switch- 
board operators, and all members of the intake staff must be sensitive to 
the impression conveyed to clients. 

2. Avoid unnecessary steps in the intake process: Some specialists maintain 
that each tune potential appUcants are required to make another visit, 
roughly half wiU not return. Recruiting experts strongly argued against the 
practice of erecting hurdles to "screen out" the unmotivated. In fact, an 
extended intake process may screen out those with other choices, leaving 
only those with no other options. 

3. Present the benefits of the program before its costs: In many SDAs, 
apphcants must provide considerable documentation to estabUsh their 
eUgibiUty for JTPA^ entailing the expenditure of time, the stress of 
providing personal information to strangers, and the direct costs of 
transportation and, sometimes, chfld care. If the beneGts are presented 
clearly Orst, potenti'al appUcants are more likely to pay the price of 
admissioij- 

SDA staff were also encouraged to let the researchers know about problems related to 
the study. While there were limits to the changes that could be made without compromising 
the research, there was some flexibiUty, and adjustments could be considered. Ongoing 
commum'cation and cooperation wrcre essential 

^- Potential Costs of the Study f nr SDAs and Their Comp^nsafinn for Participatinp 
Few studies of local programs, including those that use a noncxpcrimental research 
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methodology, can be completed without some assistance firon? program staE Salaries that 
cannot be absorbed - whether of on-board staff or new staff hired to cover additional activities 
- represent a cost of the study. Even in the least intrusive studies, staff need to assist in 
locating records and often are interviewed about the proiram. A random assignment field 
experiment requires c.en more extensive, active involvement of local staff members. This is 
needed to implement the random assignment procedures in intake and service delivery and to 
ensure that an individual's experimental/control statas is not changed for the duration of the 
post-random assignment follow-up period. 

In this study, SDAs were concerned about three kinds of costs. First, implementing the 
study procedures, especially the addiUonal recruitment efforts and data coUectfon, would impose 
direct costs on the sites. Second, they were aware that the study might have repercussions for 
performance-based contracts with their service providers. FmaUy, SDAs were concerned about 
possible future losses of incentive funds. 

Recognizing these concerns, the Department of Labor approved a shift in project 
resources that allowed increased compensaUon to the sites for their efforts in implementing the 
study. Authorized compensation increased from an average of $40,000 per SDA to an average 
of $170,000 for the im'tial and continuing costs over the approximately three-year period of the 
study.* Funds were allocated based on estimates of each site's administrative effort in 
implementing the study. Ibis, in turn, was affected by the size of the research sample, the 
number of offices and organizations involved in random assignment, and the difficulty of 
recruiting appHcants. Payments ranged from $50,000 to $235,000. Since the contracts were 
not structured as cost-reimbursement contracts, SDAs sUTl had to agree to participate in a 
study in which the actual costs were unknown and complete expense coverage could not be 
guaranteed. The increase in compensation, while helpful, was small compared to the amount 
of incentive funds some SDAs felt they vrould place at risk by joining the study. 

A. Direct SDA Costs of Implementing the Study 

The following discussion summarizes the required involvement of the SDAs' staff in the 
study's data coUecUon activities. As this shows, there was substantial effort, and the level of 
compensation had to reflect this. 



*As discussed below, the funds saved by omiuing a baseline information phone survey were used to 
increase suppon payments to the sites. 



-51- 



EMC 



• Baseline histoiy: A baseline histoiy was needed for each sample mmber, 
vsmg a form that would be standard for aD the study sites. In the original 
research design, the researchers were to obtain the history by phone (with 
in-pereon backup) immediately after random assignment The interview 
would take approximately 30 mmutes and cover family composiUon, 
^ucauon and training, wort: eqjcrience, earnings, and welfare receipt" 
However, as part of the shift in projert resources, the department approved 
subsutuuon of a less detafled form, to be completed as part of the 
ehgibihty process. *^ 

The revised form, now four pages long, could be filled out by program staff 
as tn^ interviewed the applicant or completed by the appUcants and then 
bnefly reviewed by staff for accuracy and completeness. This background 
mformauon form (BIF) had to be filled out at some point prior torandom 
assignment which meant that SDA and service piovider staff often had to 
review the BIFs for more people than were actually randomly assigned. 

• EnrolhntfnJ and tcnaination records: For the duration of the s udy the 
researchers need JIPAprogram enrollment and termination records for the 
entire sample. These are obtained from the SDA management information 
system data base. Data on both initial enroUment actions and enrollment 
and termination from various components are needed. 

• CdstRKords: Cost records maintained by the SDA are required to track 
c;q)enditures for specific types of activities. Cost data on other types of 
serwcK received by the treatment and control groups are obtained from 
published reports. 

• Intenriews and document reviews: SDA staff are interviewed and 
dCKmments are reviewed at regular intervals over the course of the study. 
Information is obtained on local economic conditions, administration of the 
JiPA program, program activities, and implementation of the study. 

OUier data sources needed for the impact analj-sis are direcUy accessed by the 
researchers, but SDAs sometimes faciUtate access to the information. These sources include 
the information collected during follow-up interviews at approximately 18 and 30 months after 
random assignment The follow-up interviews update Uie baseline informatioa^ They are 
fielded by phone (with in-person backup) and take approximately 30 minutes to complete. 

Administrative records outside Uie SDA arc an important source of information about 
the employment earnings, and welfare receipt of all individuals in the sample. Public 



ic 30-monih foUow-up interview updates infonnation coll&cted at the 18-monih intcrvii 
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assistance, unemployment insurance, tax, and Social Security records are accessed to the extent 
feasible. 

In addition to these data collection responsibilities, SDA and/or service provider staff 
were also responsible for explaining the study to applicants. They were expected to be able 
to present the project positively and clearly and to spend extra time with individuals who had 
questions. After the experimental status was assigned, staT had to follow speciCc guidelines 
in dealing with those who had been assigned to the control group. For those assigned to a 
treatment group, staff needed to ensure the provision of only those activities that were 
designated at the time of random assignment These guidelines relating to the treatment and 
control groups had to be followed for 18 months after an individual's random assignment 
Monitoring procedures had lo be put in place to assure that all this would happen. 

Potential Costs to Service Providers Under Performance^Based Contracts 

Service providers were concerned about two kinds of potential costs. First, in more 
decentralized SDAs - those in irfiich service providers played key roles prior to random 
assignment - methods for compensating the providers for the efforts discussed above were 
negotiated. Second, service providers who were reimbursed under performance-based contracts 
were greatly concerned that establishing a control group could result in their failing to meet 
their payment benchmarks. This could happen if increased recnritment did not produce 
enough additional apph'cantii, or if the service providers had to be less selective about whom 
they did enroll How to ensure that service providers were not disadvantaged because of the 
sfudy, while avoiding a "hold harmless* situation that would change the nature of the system 
in which they operated, was a dilemma similar to the issue of assurances from states to SDAs. 
In decentralized SDAs, where service providers are responsible for recruitment, SDAs needed 
to be sure the service providers had an incentive to undertake more aggressive recruitment 
activities. Agreements that gave service providers payments for additional individuals recruited 
and determined eligible (as needed to create a control group) were critical to obtaining the 
support of the contractors in some SDAs. In addition, some SDAs agreed to take info account 
the existence of the control group in judging providers' performance. They sometimes agreed 
to apply any 'adjustments" they received from the state to their contracts with service 
providers. 
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C. Poteatial Lo ss of feceativa f nttAR 

While past evaluations provided no reason to beUeve that SDA perfonnance would 
decline as a result of participating, some SDAs stiU had concerns. Hiese were best addressed 
by assurances from the state that adjustments to performance standards would be made if their 
performance declined because of participation in the study. Some SDAs requested a 100 
percent "hold harmless," but the researchers rejected this because it would change a significant 
factor of the system as it normaify operates. 

Some states were reluctant to promise that governors' adjustments to the standards would 
be made if they were necessary. The department, aware of the states' concerns, released a 
letter strongly encouraging them to support the participation of SDAs in tiieir state. The 
depa.tment also provided preliminary guidelines for adjusting the standards and expenditure 
levels for SDAs in the study. Most states did tiien indicate a willingness to provide either 
verbal or written assurances that declires in performance tiiat could be attributed to 
participation in the study would be considered. The specific assurances varied, but most states 
accepted the possibiUty of underesjpenditures if recruitment did not compensate for those 
assigned to the control group, and that outcomes on performance standards might change in 
unknown ways. In addition, adjustments for tiie enroUment of harder-to-serve individuals, 
under-enrollment of some target groups, and potential costs of participation that exceeded the 
amount of study reimbursement funds were also assured. 

V. locreasing the Benefits of the Information Provided to Study Sitps 

SDAs wiU receive detailed information on the characteristics of aU study sample members 
plus outcome data from 18- and 30-month foUow-up interviews. This is far more extetic.ve 
information, with a tonger follow-up, tiian SDAs could acquire on their own. As to impact 
Gndmgs, im'tially tiie research protocol specified t'-at SDAs could only be offered results on 
their own program if the impacts were statistically significant- The relatively smaU sample sizes 
in many SDAs meant that they would receive few results on impacts for individual target 
groups or treatments. The intensity of SDA interest in this issue caused the researchers to 
agree to provide them findings on the selected subgroups and major activity components even 




if thqr are not significant at conventional statistical levels.* 

The technical assistance and training on cUent recruiting and intake procedures also 
constituted a benefit to participating SDAs. Further, SDAs were offered the chance to 
participate in the naUonal poUcy debate at project conferences attended by senior department 
officials and leaders of the JTPA system. Fmally, the extended contact with the research team 
provides many opportuniUes for SDA staff to learn new study techmques. Attendance at 
conferences to discuss the impIementaUon of the study and findings provides an opportunity 
for staff development in study methods and data collecUon techniques. 

In summary, the general concerns were addressed by conveying the need for a rigorous 
study and by carefully modifying specific aspects of it Ethical concerns were diminished by 
shortening the exclusion period for those assigned to the control group and by aUowing a list 
of alternative services to be provided. Legal and public relations concerns were minimized by 
a written statement of the legaUty of random assignment, evidence from past studies, and 
assistance in explaining the study to groups outside the SDA, including the press. Concerns 
about the financial costs of participaUng were partially resolved by an increase in compensation 
and, in some states, assurances from the governor's office that adjustments to performance 
standards would be considered if participation in the study affected performance. An increase 
in the random assignment ratio of treatment group members to controls and technical 
assistance on cUent recruiting and retention lessened enroUment concerns. 

Hie operaU'onal concerns of sites, reviewed in Chapter 2, were primarily addressed by 
explaining particular elements of the research plan or by modifying the plan. This process is 
discussed in the next chapter. 



<^tudy sites have been briefed on the cautions that are needed when using results that are not 
statisticaUy significant. Analyses and briefings will include explanations of the degree of confidence thai 
should be aiiachcd to each of the findings. a ^ ^ mai 
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CHAPTER4 

BALANCING RESEARCH GOALS AND SDAs* OPERATIONAL CONCERNS 



Caiapter 3 discussed how the Department of Labor and the research team attempted to 
deal with SDAs' general concerns about the study. These issues had to be addressed to an 
SJJA's satisfaction or it would not consider participating in the study. But SDAs were also 
concerned about how the study's implementation would affect then- intake procedures, 
assessment practices, development of a service plan, and service delivery. These issues also had 
to be dealt with in developing a detailed study design, but they could not be considered in 
isolation. 

Sites' operational ccn&;ms had to be balanced against the several, sometimes competing, 
research goals of the project: ptoducing reliable impact estimates for the overall JTPA program 
in a sample of SDAs plus estimates for specific kinds of program activities or "treatments" and 
estimates for particular subgroups or "target groups" of program participants. The study sought 
to do aU this while changing tiie existing program as littie as possible. The unusual features 
of the JTPA nrogram and the concerns of SDAs, discussed in Chapter 2, contributed to this 
challenge. 

The decision to use random assignment - while the most important made by the 
department in planning the study - was only the first step in developing the research design. 
Other key questions, common to all research of this type including nonexperimental research, 
also had to be addressed- Doing so required tradeofS: 

1. How should sites for the study be selected and recruited? In selecting 
sites, how much emphasis should be placed on the representativeness of 
participating SDAs versus their willingness and ability to comply with study 
procedures? How should adequate sample size in the participating sites be 
assured? 

2. What types of changes in the SDAs' program and operations should be 
made as part of the research plan? How much emphasis should be placed 
on studying the program as it normally operates versus making changes that 
would facilitate implementation of the study and allow it to address 
additional important questions? 

3. Where in the intake process should the point of random assignment be 
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placed? What was the relative importance of estimating the overall impact 
of the programs studied versus estimating impacts of particular kinds of 
activiUes? What was an acceptable level of burden to place on SDA and 
service provider stafif in order to refine the impact estimates for specific 
types of activities? 

4, For which spedik Idnds of activities or "treatments" should Impacts be 
estimated? As was attempted in the CETA research, should impact 
estunates be calculated for individual types of t^Ues such as on-the-job 
trammg (OTT) or classroom occupaUonal training? Or should thev be 
calculated for more flexible categories of activiUes that corresponded to the 
usual service plans in JTPA? 

ITierG were also interactions among these questions. For example, decisions about how 
much emphasis to place on studying the program as it normaUy operates had implications for 
possible points of random assignment and for definitions of the treatments for which impacts 
were to be estimated. 

Tnis chapter uses these questions as a framework to summarize (1) the tradeofife among 
research objectives, (2) the im'U-al research plan ouUined by the department in tiie RFP for the 
study, and (3) the final plan put in place in participating sites.^ 

At the time tiie project began, tiiere was limited directly relevant experience for deciding 
how much the study could accomplish. Consequently, the department started with a plan - 
which the department and research contractors agreed was very ambitious - and envisioned 
it being modified as the researchers coUected additional information about the details of JTPA 
administration and the reaction of SDAs. This process of reconciling Uie original research 
design wiUi the operational reah'ties of the system !sd to important changes tiiat set clearer 
priorities among research objectives and addressed concerns of SDAs. 

I- Selecting Sites 

Choosing a site selection procedure required three considerations to be balanced: 

1. Representativeness: Sites should represent the diversity of SDAs in the 
system. If this were the only goal, sites would have been chosen randomly 



In the process, it provides background on the implemcnution of random assipment research 
designs for those unfamiliar with this kind of research. The details of the final research plan are 
presented m Abt A^tcs. ICF/NORC. Manpower Demonstration Research Corporation, and New 
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from among all SDAs, and participation in the study would have been • 
mandatory. Selection on this basis would have increased the "external 
vaKdity" of tiie results, that is, their generalirability to the rest of the JTPA 
system. 

2. WiUingness and capacity to Implement the research model: Implementation 
of the research model would depend on the cooperation and administrative 
abilities of tiie local sites. Those qualities would be important in assuring 
that enough applicants would be recruited to aUow creation of a control 
group without a decrease in services; that random assignment would be 
conducted properly; that those in the control group would receive services 
consistent with their treatment designation; and tiiat control group members 
would not receive JTPA services during tiie period of their exclusion from 
me program. If successfully implemented, the random assignment research 
design would produce impact estimates that would be unbiased and would 
have "internal validity." 

3. Sample size and composition: The larger the sample of people analyzed, 
Uie more precise the impact estimates. Sample sizes for key target groups 
and treatments were also a consideration in Ught of tiie goal of estimating 
these impacts. 

There were also three key constraints on the SDA selection process. First, to control 
tiie costs of tiie study, the RFP specified tiiat no more than 20 sites were to be included. 
Second, the department decided tiiat participation by SDAs would be voluntary because they 
would probably resist mandated participation and because only wiUing partidpants were likely 
to provide tiie necessary- day-to-day cooperation. Fmally, so that tiie research design could be 
implemented without major changes in the normal operation of a site's JTPA program, SDAs" 
existing procedures had to be reasonably consistent with tiie research design. 

The research team of MDRC and Abt Associates and the department had to weigh all 
these factors in developing a site selection process. Two main strategies were initially 
considered The first would emphasize quick identification of sites that were willing and able 
to implement the research model, while still seeking sites tiiat represented the diversity of the 
JTPA system. This option recognized early indications of strong concerns by SDAs about the 
initial research design. The second option placed great emphasis on the representativeness of 
sites. This approach recognized the clear advantages of a statistically representative sample of 
SDAs in drawing generalizations about the national TWA program. With a demanding 
research design and voluntary site participation, achieving a strictly representative sample was 
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unlikely, but the department felt it was worth attempting. Tlius, whfle the department and 
research team put in place the latter approach, they agreed to an early review of progress in 
site selection and possible revision of the site selection plan. 

Site selection and recruitment occurred in three phases, reflecting adjustments to the 
procedures and research design in the course of working with potential SDAs for the study.2 

A. The Initia] P robabilistic Selection Model 

In the Grst phase, the department emphasized the goal of site representativeness. Since 
partidpation in the study was not mandated, a purely random sample of SDAs was not a 
realistic optica. Instead, potential sites were designated through a "probabilistic"^ selection of 
a representative sample of sites and through replacement of sites that did not agree to 
partidpate in a way that preserved the representative character of the sample. 

Hie probabilistic selection process had several steps. Beginning with the list of aU 
SDAs, the department and research team first eliminated very smaU SDAs, those ^vith serious 
administrative or legal problems, and tiiose outside the contiguous forty-eight states. The team 
then grouped the remaining 425 SDAs into 20 categories based on size, region, and success at 
meeting performance standards.-* Twenty SDAs were chosen (one from each category) using 
a random selection process tiiat oversampled larger SDAs. An ordered list of replacement 
SDAs for each designated SDA was also aeated If tiie designated top prionty SDAs could 
be recruited, tiiat v/ould produce more generalizable research findings tiian would a selection 
process based on solidting volunteers from aU SDAs in the system. 

In August 1986, MDRC began to contact SDAs in each category about participating in 
thcstydy. They were presented with tiie original research plan. (The original and final plans 
are outlined later in tiiis diap^er and summarized in Table 4 J at its end.) Over the next five 
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^is topic is discussed in dctaiJ in Chapter 5. 

term is used because every SDA in a given category would have an equal probability of be.ne 
selected into the sample. i r j s 

"This was measured by calculating the difference between the percent of adults cnterine 
employment after the program and the performance standard calcuUtcd for the SDA using the 

a«Jj"S"n«t modeL SDAs doing better than the standard had positive 
ratmgs on this measure. ^ 
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months, MDRC staff contacted 73 SDAs^^ At the end of 1986, none of the top-priority SDAs 
had agreed to participate, although four were promising candidates,^ In eight of the 20 
categories, MDRC had contacted four or more SDAs about possible participation. 

B* The Shift to a More Flexible Recruiting Approach 

The SDA response led to major changes in the site selection process in January 1987, 
the beginning of the second phase of site selection. The goal consciously changed from getting 
a sample of SDAs selected through a probabilistic process to seeking a sample that would be 
broadly representative of the diversity of the JTPA system. Tlie department and the research 
team agreed that MDRC should contact SDAs within each size/region category without regard 
to their order on the list and should initiate discussions with several SDAs in a category 
simultaneously. 

C Modifications in the Research Desig n 

The third phase of the site selection process began in Apriil 1987, when the department 
approved the final research plan. Changes in the period of exclusion for controls, in the 
treatment categories for the analj^is, and in the compensation to sites for administrative 
expenses made site recruiting somewhat easier. However, as discussed in Chapter 5, the 
acceptance rate among SDAs never exceeded 10 percent in any period of the site selection 
process. No doubt, to some unknown extent, the "negative momentum" and adverse publicity 
about the initial research design affected the response of SDAs in later periods of site 
recruiting. 

D. A Summary of the Sites in the Stndv 

Table 1.4 lists the SDAs participating in the National JTPA Study Together, they had 
an expected sample size of nearly 30,000. However, for reasons discussed in Chapter 5, the 
final sample size was approximately 20.000. 



In addition, another ten SDAs not in the probabilistic sample contacted MDRC about the siudv. 
Substantive discussions were not held with these SDAs at this time, but in many cases MDRC 
recontacted them later in the site seleaion process. 

^Discussions with SDAs initially contacted during this period often continued for several months. 
Four of these SDAs - Nonhcast Indiana (Fon Wayne), Macon/De Witt Counties (Decatur), Illinois, 
Larimer County, Colorado; and Oakland, California - later agreed to panicipatc. 

•61- 



ERiC 



82 



Chapter 5 also reviews the characteristics of the SDAs selected through this process and 
compares them to the national JTPA system. To briefly summarize their characteristics, they 
included two SDAs &om the Northeast, four fiom the South, sevei from the Midwrsi, and 
three from the West According to informaUon for program year 1986, the year in which site 
recruiting began, they served percentages of white and minority groups, school dropouts, and 
welfare recipients similar to the average for SDAs in the "pool" from which SDAs were 
recruited for the study. They tended to be some^at smaller than the average SDA in the 
pool and to have done sUghtly better on measured perf-ormance. 

°- Stpdving t^g Proyram as f t NormaTlY Oy w^mfpc 

JTPA is an evolving program in which the statute, regulations, and administrative 
practices have changed to reflect new poUcy priorities. Future changes in JIPA are possible 
as Congres- is currently debating major reforms of the program. As a result, the department 
and research team recognized that the program would not remain unchanged throughout the 
study. And clearly, by the time the final results of die research are avaflable, tiie program will 
have changed in new ways. For tlis reason, tiie focus of tiie research is on questions that wiU 
continue to be important no matter how the program evohrts. 

At Uie same time, tiir department and research team did not want to induce unnecessary 
changes in the normal operation of the program in Uie process of implementing tiie study. 
Some changes were inevitable with tiie intraiuction of random assignment; oUiers could be 
made to faciUtate tiie implementation of study procedures. But aU possil changes had to be 
appraised in hght of tiie goal of studying tiie program in a form as close as possible to the way 
it normally operated. He department's key areas of concern included: 

• Oient recruitment, intake, and selection practices: aanges in tiiese 
practices could lead SDAs to serve different types of cUents than usual. 

• Assessment of client seeds and service recommendations: Changes in 
tiiese; iniakc steps could lead SDAs to provide different types of services 
to clients than usual The newly chosen services could have different 
mipacts trom the services normally recommended for clients. 

• Sendee dellveiy: Changing the nature, quality, or duration of the services 
offered m SDAs could change tiicir impacm. 
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• Performancs assessment: Oiangmg the SDA services were judged by 
modifying or overriding the existing performance standard system could 
lead to changes in program services (such as a shift to more costly 
activities) or less concern about job placement 

If these types of changes occurred, the impact estimates might not reflect the program 
as it normally operated, though they would be internally valid for the services provided to the 
individuals in the sample under the performance standards u effect at the time of the study7 
Such changes might also limit the usefuhiess of the project's nonexperimental research: (1) a 
study of SDAs' recruitment and selecUon processes and (2) statistical modeling of SDAs* 
selection and assessment processes to improve methods of identifying a nonexperimental 
comparison group. To the extent that the JEPA study causes changes in recruiting, participant 
selection, and assessment, the lessons from these nonexperimental studies would be less directiy 
applicable to JTPA. 

Etiier the research team or tiie sites did, in fact, propose several changes of this nature 
early in the design and implementation of the study, requiring the department to set priorities 
among the research goals. 

A. The Enrollment Problems of SDAs 

When random assignment was proposed for the JTPA study, policymakers and 
researchers assumed that the establishuient of a control group would not result in an actual 
reduction in the number of people receiving services at the study sites or, therefore, in an 
underexpenditure of funds. Since tiie national JTPA allocation provided enough resources to 
serve only 5 to 10 percent of the eligible population, it was believed that there would be many 
other equally eligible and needy individuals who would be enrolled and "take the place" of 
those assigned to the control group. In fact, random assignment, it was felt, would simply be 
another and potentially fairer method used by SDAs - in addition to their normal selection 
criteria - to allocate limited training slots among a far larger number of eUgible people. 

Consequentfy, the underenrollment and cUent recruitment problems of SDAs, common 
throughout the system and mentioned by nearly half Uie SDAs contacted about the study, 



One type of difference from normal operaUons is relatively easy to compensate for. differences 
in the kinds of clients served. Using modeling techniques and the observable charaaensiics of people 
in the sample, the calculated impaa estimates can be adjusted to reflea the composition of the usual 
SDA .client population. 
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created serious dfficuIUes for the research. Recruitment became a pressing issue with the end 
of the recession m the mid-lSaJs. Before this time, local programs often rcUed entirely on 
"walk-ins" to their program or referrals from other agencies and were unprepared to conduct 
active outreach. The NaUonal JTPA Study began as the economy C3q)anded and 
unemployment declined: In 1984, the naUonal unemployment rate stood at 7.5 percent; by 
1986, when the study began, it was 7.0 percent; and in 1987, when random assignment began, 
it was d2 percent Table 4.1 shows the change in unemployment rates in the sites during site 
selection and early implementation of the stutiy. 

This "shortage" of appUcants meant that creaUon of a control group couid cause a 
decline in the number of people served in the study's SDAs unless: 

1. SDAs increased outreach for cUents and/or enrolled more of the people 

inquired about JTPA; or 

2. the proportion of the sample assigned to the control group was kept low, 
lessening the statistical precision of all impact estimates and possibly 
precluding meaningful impact estimates for speciGc treatments and target 
groups of appUcants. 

Recognizing the centrality of this issue, boUi ethically and operationally, the department 
agreed to provide parti'cipating SDAs and tiieir ssrvice providers with extensive techm^cal 
assistance on recruitinent and retention techmques. As discussed in Chapter 3, this training 
was done by specialists in recruitment and program marketing from Cygnet Associates. 
Although it was designed to help SDAs recruit more of tiieir existing target cUents, the training 
no doubt snbUy changed the average cUent characteristics.^ 

Mos'i SDAs responded to tiie techm'cal assistance m only minor ways, modestly altering 
then- promotional materials and intake procedures. A few did more. For example, one SDA 
(SpringSeld, Missouri) streamlined the initial steps m its appUcation process and more clearly 
described tiie beaeOts of program participation before requiring cUents to complete the 
extensive appUcation fornis. 

To case enrollment problems, as discussed in Chapter 3, the department also appro*red 
a change in the random assignment ratio from one treatment group member for each control 
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TABLE 4.1 

UHEHPLOYHEHT RATES IN JTPA STUDY SITES, 
BY PROGRAM YEAR 



Unemploiment Rate 



Study Site 


I9S4 


1985 


1986 


1987 


Capital Area, MS (Jackson) 


6.7 


6.9 


6.7 


7.2 


Concentrated Employment Program, 
HT (Butte) 


9.0 


7.S 


7.7 


7.4 


Coosa Valley. GA fRome) 


7.5 


7.0 


5.7 


6.5 


Corpus Christi /Nueces Count v TX 


9.1 


8.4 


10.4 


12.0 


Cravfford/Hancock/HaH nn/Uv»nrfnf 
Counties, OH 


10.2 


10.4 


10.0 


8.3 


East Central Iowa (Cedar Rapids) 


6.2 


5.9 


5.9 


4.3 


Greater Omaha, HE 


4.8 


4.7 


5.6 


5.0 


Heartland, FL (Lakeland) 




10.9 


10.3 


9.4 


Jersey City, NJ 


in Q 


10.5 


9.6 


7.9 


Larimer County, CO (Fort Collins) 


A Q 


5.2 


5.8 


7.4 


Macon/De Wit*^ rnunff^c Tf ^nar»fiiv*^ 

rtawwii/ «/w R I bit WUUIILIO, X i> ll/cCaLUri 


10.7 


10.2 


11.3 


10.6 


Northeast, IN (Fort Wayne) 


8.9 


7.2 


6.0 


5.1 


Northwest, KN (Crookston and 
Thief River Falls) 


10.3 


9.1 


10.0 


8.6 


Oakland. CA 


7.0 


6.8 


8./ 


7.7 


Providence/Cranston, RI 


7.2 


5.2 


4.6 


4.3 


Springfield, MO 


7.7 


6.4 


5.9 


5.7 


SOURCE: JTPA Annual Status Report (JASR) for 


program years 


1984-87. 







NOTE: These years represent the labor market context for sites' decisions about 
participating in the study. 
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to two per contioL«» Fbr some SDAs, the departmem authorized esdusion &om random 
assignment of certain espedally haid-to-rccruit types of appUcants such as older woricers and 
handicapped applicants. Thus, aD the eligible membcis of such groups could be served, since 
none would be a-.igned to a control group. In addition, in a few instances the departmeni 
authorized a temporaiy increase in the random assigmnent raUo to three treatment group 
members (or. in some cases, sis) to each control to lessen recruitment problems for specific 
groups in the sample and allow an SDA to continue to participate. 

^ -Estimating the Relative mtff erentia!) Tmnarts of Treafmenfc 

Random assignment research can provide soUd evidence on what type of service works 
best for what type of appUcant, informaUon that can be useful to SDAs. However, calculating 
the relative (differenUal) impacts of specific treatments through a random assigmnent field 
study would require major changes in the way SDAs and service providers choose what services 
to provide their chents. 

Hgure 4.1 iDustrates the type of random assigmnent design that would be needed to 
determine the relative impacts of various services such as OJT, classroom occupaUonal training, 
and job search assistance. AppHcants wb. latisficd JlPA eligibiliiy requirements would be' 
randomly assigned to one of these three types of activities or to a control group, which wouid 
not be allowed to participate in JIPA. However, such random assigmnent to treatments would 
override the SDA's or service providers' usual assessment process, introducing a serious change 
in program operations. 

To compare the treatments' relative impacts, the groups in each treatment would have 
to be the same except for the difierences in treatment But in JTPA's normal operaUons, 
f^^S typically aecommend different types of apphcants to difTerent types of activiUes. Thus,' 
without random assignment to treatments, it would be impossible to know whether different 
impacts resulted from the different treatments or from differences in the groups receiving them. 
As schematized in figure 4.1, random assignmem to treatments allows estimates of each 
treatment's net impact (by comparing -Jie group receiving it to the control group) and the 



*I^is change wUl result in ao approximately 12 pftrcent decline in the statistical precision of the 
c^raU impact estimates. If the raUo had been chang.5 to three treatment group mem£^rcom Jl 

a^im " '"^"^ '^'^ ^ furthe^iSSl- 
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FIGURE 4.1 



RANDOM ASSIGNMENT TO TREATMENTS 
OR TO A CONTROL GROUP 



Service #1 




Service #2 



Service #3 



Contro! Group 
No JTPA Services 
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relative impacis of the different tieatmeats (by comparing treatment groups vnth one another). 

A less intrusive design, such as that shown in Egurc 42, would randomly assign people 
after the SDA or service provider staff recommended them for a tjpe of service based on an 
assessment of their current skills, interests, and training needs. In other u-ords, different types 
of people would be referred to each type of service. Hence the mt impact of each treatment 
could be calculated by comparing the essperiences of those assigned to it with the 
corresponding control group. But the experiences of the three treatment groups could not be 
compared, since they would differ in both treatment und baseline characterisUcs. 

Hie payoff to the more intrusive design of Rgure 4.1 is clear. But it would obviously 
conffict with the goal of studying JIPA as it normally operates by overriding the usual 
assessment process. It also would substantially increase SDA and service provider resistance 
to the study because they would be unable to taflor their services to the individual needs of 
apphcants, a flexibility they value highly, as discussed in Chapter Z For both re-isons, the 
department and research team rejected a researcn aesign that would allov- estimates of the 
relatwe impacts of treatments. Instead, as discussed later in this chapter, a design similar to 
that shown in Figure 42 was chosen.^" 

C Performance Standards 

Performance st-mdards was the third area of concern about possible study.induced 
changes in JTPA Many SDAs thought that creation of the control group, coming on top of 
recruitment problems, might force them to serve less job-ready and motivated appUcants. They 
also believed that the performance standards adjustments aUowed by the department's using 
its regression model did not adequately reflect the difficulties of serving and placing less job- 
ready applicants." For both reasons, they beUeved their measured performance and, hence, 
incentive funds would decline. 

Some SDAs requested a departmental guarantee that their incentive funds would not 
decline if they partidpated in the study. This option was rejected on two grounds. Fust, the 



<^onrxpcnraentaI research that attempts to compare the impzas of treatments faces subsuntial 
obstacles because of the problem of scleaion bias. Not only must rcscarcers successfully mcuel the 
process of sclecuon into the program, but they must also model assessment of needs and enrollment 
mto specific types of treatments. 

* See ihc discussion of performance sundards in C3iapter 2 and Appendix A. 
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FIGURE 4.2 

RANDOM ASSIGNMENT FOLLOWING 
ASSESSMENT AND RECOMMENDATION OF SERVICE PLAN 



Recruitment 



Eligibility 
Determination 



Assessment 



Service #1 



^ Reco mmended For 



Service 



Servico #3 



Random 
Assignment 



Random 
Assignment 



Service #1 



Control Group #1 
No JTPA Services 



Service ^2 



Control Group #2 
No JTPA Services 



Service #3 



Control Group #3 
No JTPA Services 
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states play the central role in developing formulas for the distribution of incentives funds and 
resolving disputes over measured performance. In this decentralized system, the department 
tvas not in a position - administratively, poUUcally, or possibly legally - to override existing 
state performance assessment systems. Second, the department (with the support of the 
research team) beUeved that any "hold harmless" guarantee would fundamentally change the 
way JTPA operates. In an effort to address SDA concerns, the department did issue a letter 
to state JTPA directors urging them to administer the performance standards system flodbly 
for SDAs in the study and to mske adjustments in the level of performance standards if an 
SDA made a plausible case that participaUon affected its performance rating.l2 j^t 
department also urged that if SDAs could make such a case, they not be required to provide 
extensive documen.^tion. Despite these moves, many SDAs continued to be seriously 
concerned. 

nie final importance of performance standards adjustments will not be known until mid- 
1990, the end of the program year in which random assignment was completed. In the Bisl 
two years of the study (program years 1987 anJ 1988). only five SDAs requested a special 
adjustment to meet their performance standards because of declines arguably related to the 
study.J3 p-oyj. jjjg ^^^^ ^^^p adjustment** 

To summarize: In the three k-:^ areas just discussed, the proposed major changes were 
rejected (random assignment to a treatment and "hold harmless" provisions on incentive funds) 
and other changes were marginal (changes in recruiting and suggestions from the department 
for flexibiUty in administering perfonnance standards). The department and research team 
were able to develop a research plan ^uat made no or only minor changes in (1) SDA and 
service provider staff decisions on whether a client was appropriate for JTPA (2) the process 
for recommending ervices lot apph-cants,l5 and (3) the nature and duration of the seivices 



12 

Under ihc cjdsting performanoe sianiwd system, govemon have ihe authority to make 'eovemors' 
a^tments" to reflect special circumstance i. an SDA. Sec National Assodation orSum°« eTaT. 

JoJ^Tk^^ were Corpus Christi/Nueces County. Texas; Capital Area (Jackson). Mississippi; 
^4?,^ ^"^^ ^"'l Providence/Cranston. Rhode Island 

«:nA. I °°' tcquesicd adjustment, partly because the state has only three 

SDAs so mcreasing the incentive funds for one SDA would strongly affea the other two. 

As discussed later in this chapter, the requirement that service pUns be left unchanged after 
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provided to those esrollcd 

This is not to say that participaUon in the National JTPA Study was "business as usual" 
• for the sites. The worsening reciiiitment problems, due in part to continued low 
unemphjyment in the late 1980s>, coupled with the creation of a control group, meant that both 
operating JTPA and iraplemenUag the study procedures presented challenges. From the sites' 
perspeciive, the study was one of several reasons why they might enroll appUcants less skiUed 
and job-ready than those served in the past 

^ SDA-Wide Program Fsti mates and Treatment-Spedfig Estimates 

The department's RFP for the project requested estimates of the impact of the overall 
JTPA program in participating SDAs and of speciSc types of activities. The characteristics of 
many JTPA programs and the requirements of random assignment research created a tradeoff 
between these two types of estimates, which had to be resolved in the research design. The 
basic decision for the department and research team was at what pomt in the SDAs' intake 
and assessment process to place random assignment The tradeoff impUcit in this decision can 
l)est be explained through an example. 

When people apply for JTPA activities, SDA or service provider staff typicaUy start by 
informing them about the types of services avaflable and detennine whether those who art 
interested are eligible for the program.** Staff then assess the skills, interests, and training 
needs of those eligible as a step preliminaiy to developing a plan of recommended services." 
In many SDAs approached about the study, staff reported that they provide a Uttle instruction 
in job search techniques as part of this up-front orientation prior to completing the assessment 
and developing the service plan.*^ 

On completing the assessment, staff determine if available JTPA activities are 



In most SDAs, the SDA subcontracts out much of service delivery to other agendes. In some 
SDAs, which greatly decenualizc operaUons, this includes client recruiting and intake. In these 
decentralized SDAs. the service providers would be involved in explaining the study and conduainE 
random assignment 

"This assessment process can range from an interview with a program intake worker to several 
days of testing, interviews, and career counseling. 

'^me SDAs also use it to gather further information on applicants' motivation, job-readmcss 
and ability to interview with potential employers and draw on that information in making the 
assessment decision. 
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appropriate for the applicant If so, they iscommend a service plan, often called an 
emplqyability development plan or EDP. Next, applicants are referred to possible activities: 
• an education provider for basic education, a training agency for classroom training, an employer 
for an OJT, or job search assistance provided by the SDA or contracted out to another agency. 

One opUon for the stucfy was to conduct random assignment immediately after appUcants 
were determined eligible, as shown in Rgure 43. In this case, a comparison of the treatment 
and control groups' post-random assignment experience would have provided an estimate of 
the overall impact of program services for those found eUgible. It would not, however, have 
provided experimental estimates of the impact of specific types of activiUes.^* 

Alternatively, random assignment could be done after staff assessed the skills and needs 
of applicants and identified those appropriate for particular categories of activities, as shown 
earUer in Rgure 42. The impact of having access to a specific service could be estimated by 
comparing the post-random assignment experiences of those in that service to the control 
group that had been recommended for the same service. Furthermore, comparing the 
experiences of everyone in tlie various treatment groups to everyone in the various control 
groups would provide an estimate of the program's overall impacL^o 

No serious analytical problems would arise if the assessment and counseling services 
provided before random assignment were insubstantial relative to all tfie services provided to 
the treatment group. In this case, delaying random assignment to allow impact estimates for 
categories of activities would cause only a small downward bias in the impact estimates. 

More serious problems would arise, however, in two cases: 



As discussed earlier in this chapter, applicants' current skills and job-readiness a£fect the type 
of services recommended; those recommended for OJT, for example, arc likely to be more job-ready 
tnan those recommended for basic education. Therefore, comparing the post-random assignment 
experience of treatment group members recommended for OJT (a relatively job-ready group) with the 
experiences of the entire control group would overestimate the impact of OJT. to general, under this 
desigti, It would be inappropriate to compare the experiences of individuals referred to specific types 
* a^^^f^ experiences of the entire control group, 

^trictly speaking, this would be true only if aU possible aaivities were included in the study and 
aU mdividuals recommended for activities were subjea to random assignment. This type of limitation 
on study results was present in Uie initial research design, as discussed in the next seaion of this 
chapter. Even m Uie final design, some components of Uie Title IIA program in an SDA may have 
been excluded from tiie study because of Uie operational difficulties of including them. Examples 
negotiated m one or more SDAs include transfers from programs funded by other titles of JTPA and 
referrals from oUier agencies to which Uie SDA is contractually obligated (for example, when the SDA 
ts a service provider under state welfare employment programs). 
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FIGURE 4.3 



RANDOM ASSIGNMENT FOLLOWING 
EUGIBIUTY DETERMINATION 




JTPA 
Services 



No JTPA 
Services 



er|c 



L Pre-randoBi fissi£iuaent services are sobstanttaL This could occur if 
assessment involved esrtensive testing, counseling, and job search instruction. 

2. A iHJSt-raiMlcmKslgnaentSreatinent involves kw-lntensilyservi^ niis 
could occur if the primary SDA service was job search assistance. 

In both cases, a signiOcant proportion of aU JIPA services comes before random assigmnent 
and is thus avaflable to the control group. The comparison of the experiences of the 
treatment and control group would underestimate the impact of the entire JTPA program. 
From the perspective of the agency conducting the assessment and recommending services, 
there were also competing operaUonal consideraUons in choosing the point of random 
assigmnent On the one hand, the longer random assigmnent was delayed, the more 
information staff would have about cUents' skills and interests and the more confident they 
could be about their service recommendation - but the more people the agency would have 
to -process" through its intake steps. On the other hand, if controls were identified early, they 
would not continue through succeeding stages, lessening both the administrative effort and the 
burden on clients before they learned if they coald be served under .HPA. EarUer assigmnent 
would also reduce the stress on the staff; who would not become as emotionaUy committed to 
those who were eventually assigned to the control group.21 

•nie im-Ual and final research design for the JTPA study placed random assigmnent after 
assessment to make possible impact estijiates for activities or. more precisely, categories of 
activiUes. as discussed later in this chapter.^^ In adopting this plan, the departmem chose to 
approve several sma^ changes in normal program operaUons, if needed and accepted by 
participating SDAs, to fa^oh'tate implementation of the program model These included limiting 
up.£rom job search assistance to a few hours in one SDA (Northeast Lidiana) and requiring 
SDA and service provider staff in two SDAs (Larimer County, Colorado, and Oakland. 
Califorma) to designate recommended services somewhat earUer than they had in the past 
TTie procedures for conducting random assignment were relatively straightforward, as 



had act^lL K T ^""^ assignment: to delay it until the service provider 

f^^.^.^ f ^ r«^"".ent and condun their own 

5^ bu?d<.n T!^A J?"^"^ """P'"" '° implement and increased administrative and 

a^lc^r^ro Lit , ^ ' °' P'^'-'^'-'^S --^^ ^'^^ information to use in 

fenJL R ''^V'"' ^crcasmg the participaUon rate of those assigned to the treatment group 
Appendix B includes a dcscnpUon of the random assignment procas in each ^DA. 
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shown in Hgure AAP (In this chart, the ortra steps added to the nonnal intake procedures 
because of the study are indicated by dashed lines.) The staff of the agescy conducting intake 
for the program (which, again, could he the SDA or the service provider) determined 
appUcants' eUgibiUty for the program and briefly informed them about the study, random 
assignment, and its procedures. As part of the intake process, apph'cants filled out a 
background information form, providing information on their characteristics and previous work 
and training. 

They were informed again about the study and given an opportunity to ask questions.^^ 
Many SDAs decided to use a videotape prepared by the departiaent and research team, which 
described the goals and procedures of the study, to make sure that aU appUcants got a clear, 
accurate description of the project and random assignment. Some SDAs chose to make their 
own videotape to include local scenes and famUiar types of clients. Once the study had been 
explained, applicants were asked to sign an informed consent torn? indicating that they 
understood the study procedures and gave the research team access to administrative ^r -rds 
pertaining to their earnings and welfare receipL^^ Staff then conducted their usual assessment, 
determined if the applicant was appropriate for JEPA, and, if so, recommended specific types 
of activities and designated a treatment category. 

Once staff designated a treatment categoiy and were ready to refer an applicant to an 
activity, they caUed MDRC on a toU-free number and provided the treatment categoiy 
designation and simple demographic information on the applicant (to allow mom'toring of 
sample buildup by target group).26 Hiey were then told the results of random alignment 
Those randomly assigned to the treatment group were eUgTble to receive the service designated 



precise sequence of steps varied among the SDAs to accommodate differences in their usual 
uiiake procedures. In addiUon, in some SDAs, service providers played a greater role in client 
recruf ment and assessment than that ponraycd in the text and therefore were more involved in random 
assignmenu The steps described in the text were typical of most SDAs. Sec Appendix B for a 
summary of departures from this nonnal sequence. 

^In most SDAs, applicants heard about the study several times before random assignment and 
had several opportunities to ask questions. 

^^hidividuals refusing to participate in the study and sign the informed consent form were not 
serv«^ unde/ JTPA in participating sites while random assignmeni was part of the intake process, 
v-alls could be made for individuals or batched for groups of applicants. 
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FIGURE 4.4 

iPOM ASSIGNMENT PROCEDURES 
IN THE NATIONAL JTPA STUDY 
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fay staff and were to be referred to appropriate agencies.27 itose in the control group were 
told that they could not be served in any JTPA-funded activity withic the SDA for 18 months 
and were provided with a list of other services in the community.^s 

The research team developed procedures to assure that the results of random alignment 
were followed in subsequent service deliveiy. SDA and service provider staff were instructed 
to note each person's assignment status anc: to follow that designaUon as they referred cKents 
to services. (The next section of this chapter discusses the deSm'Uon of the treatment 
categories and the restrictions on services they imply.) 

Each week, every SDA and service prmider ofBce invoked in cKent intake received a 
cumulative list of aU persons randomly assigned in the SDA and their assignment status. Staff 
involved in cKent intake were instructed to consult these lists early in the intake process to 
identify persons v^o had already gone through random assignmenL^^ They were to continue 
consulting these lists after random assignment ended untfl 18 months after the final random 
assignment in their SDA^ 

If "repeat" appKcants do reapply in the 18 months following the date of their random 
assigmnent, they are reminded of the study and their assigmnent status. Those who had been 
assigned to the treatment group could be provided services consistent with the original 
treatment designaUon. Controls were again provided with the list of other services. 

IV". Defining Treatment Categ ories 

nie RFP for the study called for impact estimates for three speciGc acUviUes: OJT, 
classroom occupsUonal training, and job search assistance. (When the study began, 
approximately 75 percent of all JTPA Title HA parUdpants were enrolled in one of these 
three activiUes.) This focus on individual activiUes was consistent with the prior. 



27r 



Delaying random assigmnent until sia£f were ready to refer applicants to an activity had the 
funher advantage of raising the proporUon of the treatment group who would aaually pamapate in 
program aamties. This point is discussed later in this chapter. ^ 

As discuss*^ in Chapter 3. the imiial research design excluded controls from JTPA for 30 months 
and did not provide a list of other services. 

^^If staff did not notice that an appUcant had already been through random assienment the 
comj^ter program used to condua random assignment would identify them as a repeat. 

At this point, the service exclusion for the last control will end and rcstriaions on service choice 
for those assigned to the group to be served will also end. 
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nonc^>erimental research done on CETA. vMch reported impact estimates for these and other 
activxties. 

Hie operational reality of the JTPA system, coupled with the requirements of random 
assignment research, led the department and research ttcm to use a different definition of 
treatments, one designating treatment -categories" based on the service plans developed for 
applicants. This shift away &om estimates for individual activitiss was made for several 



reasons. 



Even though more than 75 percent of all participants are active in only one service,^! 
at the time a service plan is developed staff often recommend several possible services, lie 
actual services in which individuals participate are determined by many factors: their interests 
and needs, the avaflabiUty of funding. opv.nings in training agencies, possible job placements, 
and the availabiUty of employers for OJT positions- This makes it difficult for SDA or service 
provider staff to choose a single service v^en they develop the service plan. 

For treatment-specific impacts to be estimated, however, staff making service 
recommendaUons must designate a treatment category pnor to random assigmnent so that a 
control group/or each treatment categoty can be identified.32 If the designaUon instead follows 
random assignment, it would be partly based on informaUon and insights gained after random 
assignment, through working with an appUcant - information that would not be available about 
control group members. For simflar reasons, the designation of a treatment category cannot 
be changed after random assignment 

Also, as noted earUer, SDA and service provider staff value their abiUty to develop 
•individualized" plans for appUcants. They felt that forcing them to choose a narrowly defined, 
binding treatment category would change their normal operaUons. even if in most cases they 
could readily comply with the service restrictions impUed by their choice of category. Thus, 
more flexible, broadly defined treatment categories were seen as more consistent with the goal 
of studying JTP/ as it normally operates. 



I'^r ^ ^ °^ performance standards on costs per person placed in a job and 

per youth for whom a positive terminaUon' is achieved. Sec the discussion in Chapter 2 and Appendix 

^As discussed above, different types of applicant arc recommended for OJT. classroom trainine. 
u^^l • " """^^ ^ ""^"^ ^° """"'^ ^^'^ °" ^P^"^ of each 
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For all these reasons, the department and research team developed a new conc^t of 
"treatment categories" linked to the service recommendations of program stag! Rather than 
providing estimates of individual activities, as was done in the CETA research, this study uses 
treatment categories that aUow combinations and sequences of activities and are consistent with 
the usual practices of many SDAs. 

Discussions with SDA staff allowed Uie research team to identify a set of categories Uiat 
would accommodate any conceivable service recommendation. Two were based on tiie most 
important components provided in JTPA - OJT and classroom occupational ti^ining^^ - while 
the third was a residual category, included to allow an estimate for the entire JTPA program 
in each SDA. Table 42 shows Uie definitions of the tiiree categories, and Table 4.3 presents 
the guidelines given to staff about using the categories. 

So Uiat impact estimates could be -ctten for service plans based on OJT and classroom 
training, tiie category anchored on OJT (to be used when recommending a person for an OJT 
alone or in a sequence) allowed any activity except classroom occupational training. Hie 
category anchored on classroom occupational training (to be used whf.n recommending a 
person for classroom occupational training alone or in a sequence) allowed any activity except 
an OJT.^ 

The 'oUier services" category was unrestricted, allowing inclusion of all possible service 
recommendations. Hiis category, capped at a percent negotiated witii the SDA, was intended 
to be used when staff (1) wished to begin a service sequence with something other than OJT 
or classroom occupational training and were undecided about subsequent services; (2) wished 
to combine classroom training and OJT, as is done in employer-customized training; or (3) 
>vished to recommend a service plan that did not invoke eiUier classroom occupational training 
or Orr (for example, job search assistance only).^' Among the SDAs in tiie study, the ceUing 



^^ta from the National Job Training Longitudinal Survey (JTLS) available at the beginning of 
the study indicated that these two activiUes accounted for approximately 40 to 50 percent of aL JTPA 
parti^)aDis. This has since increased to about 60 percent 

Eventually, the research team agreed to allow SDAs to provide a small amount of OJT following 
cL-issroom occupational training in cases where it was needed to secure placement m a job. This OJT 
could not amount to more than 20 percent of total Gaining hours. Similarly, SDAs were allowed to 
provide a small amount of classroom training (again, no more than 20 percent of all hours) when 
necdwi to give the appUcant a 'skills brush-up* in order to secure an OJT. 

SDA staff were required to list individual aaivitics they were recommending for a person ^emg 
randomly assigned. This will allow us to determine if a large portion of the people in this group were 

(continued...) 
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TABLE 4.2 

ACTIVITIES AVAILABLE TO APPLICANTS ASSIGNED TO EACH TREATMENT CATEGORY 



Assigned 

Treatment Category 



Activities Available to Applicants 



On-the-Job 
Training 
(OJT) 



CI assrocxn 
Training in 
Occupational 
Skins (CT-OS) 



Combined OJT 
and CT-OS 

(Customized 
Training) 



Basic 
Education 



Job Search 
Assistance/ 

Job 
Placement 



Kork 
Experience 



Other 
Activities 



I 

00 

o 
i 



On-the-Job Training Yes 



Classroom Training in 
Occupational Skills No 



Other Activities® Yes 



No 



Yes 



Yes 



No 



No 



Yes 



Yes 



Yes 



Yes 



Yes 



Yes 



Yes 



Yes 



Yes 



Yes 



Yes 



in 



Yes 



NOTE: ^This category is intended for applicants recownended for activities other than oiT or r^n^^^^,^ ♦ 
.ppHcnts recoo^ended for toth OJT and classroom training. There is a ceilin t o L of a anS ? S Tor 
this category; the limit is negotiated individually with each SDA. applicants eligible for 
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TABLE 4.-3 



SUIOELINES FOR SDA AND SERVICE PROVIDER STAFF 
FOR PICKING TREATMENT CATEGORIES (ACTIVITY GROUPS) 



DO HOT PICK AN ACTIVITY GROUP TF i 

You have not yet decided what activities vfould be proper for the applicant and you 

Zl IlnfiJ^Kl""^"'*-"'''""'^'""' ""^^1 have decided what 

you think would be appropriate for the applicant before picking an activity group. 

PICK THE OJT ACTIVITY GROUP IF ; 

1. You recomnend the person for an OJT opportunity; 
OR 

« « ^* You recomtend the person for sore other activity first but you plan to get the 

^Tt Z^ l '''''''' ''^ =^«»">^ ^"'"'"9 occupational skills 

and It roust be likely that the follow-up OJT will happen. 

PICK THE CLASSROOM TPATH THG IN OCCUPATIONAL SKILLS ACTIVITY GROUP TP - 

1. You reco(!r.iend the person for classroom training in occupational skills; 
OR - 

2. You reconinend the person for some other activity first but you plan to place the 

S^r^ T''^ «^'vity cannot be OJT 

and It roust be likily that the follow-up classroom training in occupational skills will 
happen. 

PICK THE O THER SERVICES ACTIVITY GROilP TC . 

1. You recoaxnend the person for scmethipg other than classroom training in 
occupational skills or OJT: examples include work experience, basic education/job search 
assistance, and vocational exploration programs; 

OR 

2. You reconiner^ the person for a special combination of both CT-OS and OJT fas in a 
customized train.ng program for « certain employer) or • planned sequence of classroom 
training in occupational skills followed by OJT, or vice versa. If the customized training 
is primarily classroom training in occupational skills, the applicant woull be included in 
the classroom training in occupational skills group. If the customized tra.n'ng is 
primarily OJT. the applicant would be included in the OJT group. But if the recomended 
service is truly a combination of the two. the applicant would go into the "other services" 
group; 

OR 

3. You have decided that the applicant is someone you would want to try to serve and 
you have decided on a first activity other than classroom training in occupational skills o-- 
OJT but you have not figured out a likely sequence of further activities 
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on the percent of the sample in the "other services" category ranged fiom 15 to 70 percent 
In an effort to avoid changing service delivery patterns, the ceilmg in an SDA ^ negoUated 
to reflect the pattern of service participaUon by enroUecs in previous years. SDAs that 
typically provided a high percent of their participants only job search assistance or special 
services other than OJT or classroom occupational training were aUowed high percentages in 
other services. 

While these Onal definitions of activity categories were flexible enough to allow staff to 
make tiieir usual service recommendations and to include all possible sendee plans, they do 
represent a potenfa-al challenge in tht analysis. Their flexibility raises the possibility of ending 
up witii service patterns for treatment group members in the three activity categories tiiat are 
difficult to describe simply. While past research on CETA he, many problems, as discussed 
in Chapter 1, the activities studied in the research were simple and easy to describe. In the 
National JTPA Study, tiie treatineat categories may be consistent witii the operational reality 
of the program but harder to describe. Furtiiermore. in what would be the worst case for 
analytical purposes, tiie differences between the actual service patterns in tiie two primary 
categories could be smalL Since tiie two primary categories (OJT and classroom occupational 
training) aUow less intensive activities as a substitute for tiie primary intensive activity (not just 
in addition to it), it is possible tiiat a substantial portion of tiiose in the treatment group in 
each category could end up getting only job search assistance or some otiier less intensive 
service. However, tiiis commonly occurs in tiie normal operation of JTPA 

While the final resolutior of this issue must await later reports, some early information 
is avaflable. As part of its mom'toring effort, MDRC coUected JTPA enrollment data on a 
sample of people in tiie study. These data provide some indications of patterns of enroUment 
in JTPA by treatment category.^*! These data show that in the calendar quarter of random 
assignment or tiie foUowing quarter, approximately one-half to two-tiiirds of tiiose in the 
treatment group enroUed in JTPA. (The actual number showed seasonal variations and 



^(.-continued) 



being rccomincDdcd for a single type of activity. If so. tlie sample size for this aaivity may be large 
calcuUtion of its separate impacts. 

i« cn?',*^.!f ^'^»''«l-^A«°ro>ln'«' records for a sample of approximately 1.000 people in aU 
16 SDAs in the study. SDA enroUment dau for the entire sample will be used in lateVrcports to 
examine this issue. 
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variations across the sites.) No one in this early sample enroUed in any activity inconsistent 
vnth his or her activity categoiy designation. About two-thirds of those in the classroom 
occupatioflal trdning categoiy who enrolled were participating in classroom traimng. However, 
for the OJT category, only about half of those who enrolled received an OJT. Placement in 
an OJT is particularly compUcated because it requires finding an employer offering the type 
of training desired, negotiating an agreement, and convincing the employer to hire the 
applicant Hiis proportion of enroUment in an OJT - which reflects the normal operations 
of JTPA - is simflar to that found in previous random assignment studies of OJTs for welfare 



recipients.^^ 



V. Summary of the Research Plan 

Table 4.4 summarizes the initial and Gnal research designs for the experimental analysis 
in the National JTPA Study and Rgure 45 depicts the random assignment model The basic 
goal of the study - estimates of the overall program and specific treatments in the participating 
SDAs - will be achieved, without major changes in the operations of the program except for 
the added recruitment effort required to create a control group. The treatinent categories 
used in the analysis represent a new approach to this type of research and are based on the 
operational reality of the JTPA system. The final research design, from the perspective of the 
sites in the study, represents an increase in tiie benefits of participating and some reduction 
in the costs. 



^'Thcsc studies found positive impacts for OJT programs despite these low participation rates 
Sec Auspos, Cave, and Long, 1988; Frcedman, Bryant, and Cave. 1988. 
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TABLE 4.4 



RESEARCH ISSUES IH THE INITIAL AND FINAL DESIGN 



Design Feature 
Site Selection 



Key Areas of the Program 
7o 86 Changed as Little 
as Possible 



Dptlons Offered to 
Neflfibers of the Control 
Group 



Coverage of Treatment 
Categories 



Point of RandcM 
Assignment 



Initial Design 

Attempt to recnift a probabilistic sainple of up to 2D 
SDAs. stratified by region and size and oversawpllng 
larger SDAs. Participation by SDAs Is voluntary. 

To the extent possible, no aspect of the program 
should be changed. 



Excluded from JTPA for 30 months following random 
assignwant. Eligible for other services In the 
coc?aHjn1ty not funded by JTPA, but not provided with 
any referrals to those services. 



Title IIA applicants who were adults or out-of-school 
youth and who were not reconnended for DJT, classroom 
occupational training, or Job search assistance were 
excluded from the study. Approximately 75 percent of 
national participants In JTPA received one of these 
three services. 



Random assignment to occur as close to the potential 
start of services as possible. Staff in complete 
eligibility determination and assessment* develop a 
service recofrwendotlon. Identify a possible service 
"-ovider, and then conduct random assignment for the 
llvldual. 



Final Design 

Recruit a sarple of up to 20 SDAs that represents 
the diversity of the JTPA systeoi, with Urge SDAs 
overrepresented. Participation by SOAs Is 
voluntary. 

Service recofj»»endat1ons, program services, and 
performance standards should be changed as little 
as posflble. Expand client recruiting as needed^ 
to penal t creation of the control group without 
reducing service levels. Change intake 
procedures, if needed, to establish a point of 
random assignment that will allow estimation of 
overall and activity-specific Impacts. 

Excluded from JTPA for 18 months following random 
assignment. After 18 months* can be served If 
they reapply, but no special outreach to controls 
permitted. Eligible for other services In the 
cowminlty not funded by JTPA and provided with a 
list of agencies but no specific referrals to 
services. 

All Title IIA applicants who were adults or out- 
of-school youth were included unless excluded from 
the study through negotiations with the SDA or 
through the SDA*s use of small numbers of 
discretionary exclusions. Negotiated exclusions 
Included programs with severe recruitment problems 
or those serving especially needy clients. 

R«hdem assignment to occur after staff complete 
eligibility detemlnatlon and assessment and 
develop a service recowaendatlon. 



TABLE 4.4 (continued) 



Design Feature 



Sample lite ^nd 
Disaggregated Impact 
Estimates 



Initial Design 



Up to 30.000. with impact estimates for target groups 
(adult males, adult females, minority youth. Z7i6 
white youth) and treatments (OJT, classroom training, 
and Job search assistance). 



Randcei Assignment Ratio 
of Treatment and Control 
6roups 



Half of sample randomly assigned to treatment group 
and half to control group. 



I 

OP 

Kn 
I 



Options for Those 
Randomly Assigned to the 
Treatment Group 



Definition of Activity 
Categories 



Eligible to receive services consistent with the 
activity category for which they were recowwended. 
Did not attempt to randomly assign applicants to 
different treatments because this would override 
normal assessment and service recofmtend^tions. 
leading to major change in program operations. 

Three categories based on the most coflmon individual 
services: OJT. classroom training, and Job search 
assistance, Cofltbinations and sequences not expected 
to be co(?n)on« 



Data Collection Kethods 



Baseline interview for 30 minutes fielded by phone 
(with in-person backup); follow-up surveys for 30 
minutes fielded by phone (with in-person backup) at 
18 and 30 months after random assigrmnt; JTPA 
program participation and termination records; cost 
records on program services; interviews with program 
staff; and. to the extent feasible, public 
assistance, unemployment insurance, tax, and Social 
Security at^inistrative records. 



lOS 




Final Design 



20.606. with impact estimates for target groups 
(adult males, adult females, nincrity youth, and 
white youth) and treatment categories based on 
OJT, classroom training, and other services. 



Two-thirds of sarnie randomly assigned to 
treatment group and one-third to control group. 
Some tetj^orary changes to a 3:1 or 6:1 ratio 
allowed in cases of severe recruitment problems. 

Sane. 



Two categories based on OJT and classroom 
training. A third category of "other services* is 
a residual for those not recofpiaended for either 
OJT or classroom training or recommended for both 
of these services. Combinations and sequences of 
activities are expected to be frequent. 

Background Information Forw filled out by 
applicant and reviewed by SOA staff prior to 
random assigrmnt; follow-up surveys for 30 
minutes fieldod by phone (with in-person backup) 
at approximately 18 and 30 months after random 
assignment; JTPA program participation and 
termination records; cost records on program 
services; interviews with program staff; and. to 
the extent feasible, public assistance. 
unemployTwnt insurance, tax, ar.d Social Security 
administrative records. 
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TABLE 4.4 (continued) 



Design ftature jnltui Design 



HoneUry Support for Average of $40,000 per SDA to defray some of th« 

Participating SOAS adifnistrativa costs of is^leisenting thft stud/. 



SOA Responsibilities Cocply uith the randca assignaent procedures in 

intake and service delivery; exclude controls fro« 
OTPA services for 30 ponths folloving randoa 
•ssignMent; assure an adequate nuaber of applicants 
to alloff creation of e control group without reducing 
the niBbtr of people served; meet negotiated sample 
size; provide the designated JTPA services to as aany 
individuals in the treateent group &s possible; and 
assist the research teaa in collecting infonMtion on 
prograw operations, participation in JTPA, and costs. 



00 
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Final Design 



Average of $170,000 ptr SOA to defray 
acfainistrative costs of isplesenting the ttufiy. 
Pay»ents based on saaple size and administrative 
effort involved in ii^>leaenting the research 
design. 

Coldly vith the randca assignaent procedures in 
intake and service delivery; exclude controls froa 
JTPA services for 18 aonths following randoa 
assignsent; assure en adequate nusber of 
applicants to allow creation of e control group 
without reducing the nueber of people served; eeet 
negotiated t^U size; provide the designated 
JTPA services to as wny individuals in the 
treatment group as possible; and assist the 
research teas in collecting information on prograa 
operetienst participation in JTPA, and costs. 
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CHAPTERS 

THE SITES SELECTED FOR THE STUDY 
AND THE RESULTING SAMPLE SIZE 

This chapter concludes the discussion of the site selection process. It begins by 
providing a more detaDed analysis of SDAs' concerns and other factore affecting their decisions 
about participating in the study. Then it addresses the issue of the representativeness of the 
SDAs in the study by comparing the sites in the sample to SDAs nationally on measurable 
characteristics. As the discussion shows, many of the same concerns about the possible effects 
of the study on local operations emerged across all SDAs, there is no simple relationship 
between local conditions and parUcipation or nonparticipation in the study. However, since 
the SDAs in the study were not chosen randomly, there are inevitably some differences 
between the SDAs that accepted the offer to participate in the study and those that refused 
or were never asked. Fmally, the chapter presents the sample sizes for the entire sample and 
for target groups and treatment categories and discusses the reasons the sample fell below the 
original ceiling of 30,000.^ 



I- An Analysis of the R^ ponse of SHAs Contacted About the Study 

Under the ground rules established by the Department of Labor, SDA participation in 
the study was voluntary. This meant that despite initial efforts to recruit a probabHistic sample 
of SDAs into the study (as described in Chapter 4), the Onal decision about participation 
depended on the local (and, tc some extent, state) reaction to the study.^ Analysis of this 
decision is important because it might yield lessons for future studies seeking to recruit local 
programs into a complex random assignment field study. It also provides insights into the 
differences between participating and non-participating SDAs, lending a context for interpreting 
later impact results from the study. 

As discussed in Chapter 4, there were three distinct phases of site selection: Phase 1, 



The charaaerisiics of individuals in the sample (such as race, sex, age. educaiion. etc) - and how ihev 
com^^ 10 aU those served in JTPA - will be analyzed in detail in a repon to be produced in late 1990.' 

state rcaaion was imponanl if SDAs were concerned about the possible impaa of the stud> on 
their receipt of incentive funds under the performance standards system. State flexibility on this lisue could 
affea the local response. 
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ftom October 1986 to January 6, 1987, involwrf a -probabilistic- selection process; Phase 2, 
beginning January 6, 1987, aDowed for the recruitment of a diverse group of sites based on 
size, region, and other characteristics; and Phase 3, &om April 30, 1^. through September 
1988, began with a modiScaUon of the research design to address SDA concerns. Table 5.1 
summarizes the characterisUcs of the sites recruited. Appendix B provides addiUonal detail .3nd 
a profile of each site in the study. 

Table 5.2 shows the response of all SDAs with which MDRC staff had substanUve 
discussions about participaUng in the sUidy.3 Uns information is presented in two ways: The 
upper panel groups SDAs according to the dat.. of MDRCs im'Ual contact wiUi them, while 
the lower panel groups them according to when they made a final decision about participating 
Both perspectives are useful: SDAs' reactions were often heavily influenced by tiie research 
design that was in place when tiiey were im'Ually contacted, but discussions often continued for 
an extended time and tiie terms of participation could change. 

From botii perspectives, the acceptance rate increased somewhat over time, but it 
remained low tiuoughout tiie entire site selection process. As tiie top line of the upper panel 
shows, during Phase I of site selection, MDRC was in contact with 83 SDAs.* Five (6 
percent) eventually paru'cipated, while 56 rejected tiie offer, and 22 were dropped by tiie 
research team and the departinent as inappropriate for tiie study.^ In Phase 2, MDRC 
contacted 61 additional SDAs, of which 6.6 percent eventually participated, while in Phase 3 
MDRC contacted 85 additional SDAs. 82 percent of which participated. During Phase 1, a^ 
shown in tiie top line of the lower pane!, 48 SDAs made a final decision about participating 
in tiie study based on tfie research design in place at that time; all but one chose not to 
parti-dpate. The acceptance rate of tiiose SDAs tiiat dedded in Phase 2 was 4.9 percent, and 
in Phase 3, it was 10 percent 

Two points must be kept in mind in reviewing tiiis table. Fus% in many SDAs, aU major 
agendes and offidals seriously affected by tiie study had to support tiie effort: One strong 



^^n,;*!t5'^!??K'°'''S'n? ' °f ''»^y but never responded. 

a sm!lt^r'^Ti}^f^ "^"^ '^'li" °^ admiaisirative reorganization: l^.ey were facing 

Se^^d^-^^'?hr°'' P"'"'"^^ P™^*" configuraiion could noi be aiomraoda.^^ 
wjuun the research design, or they were too geographically dispersed or served too few peep'- 
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TABLE 5.1 



SELECTED CHARACTERISTICS OF SDAs PARTICIPATING IN THE NATIONAL JTPA STUDY 



SDA 


Region 


Size 


City 


Target 
Sample 


Capnnl Area, HS (Jackson) 


South 


Small 






Concentrated Employment 
Program, HT (Butte) 


West 


Mi^di tin 

new 1 uiii 


dU t 


OOP 

8Z5 


Coosa Valley, GA (Rome) 


South 


Medium 


Rome 


1,800 


Corpus Christ! / 
Nueces County, TX 


South 


ncu 1 uni 


Lorpus Lnristi 


1,500 


Crawford/Hancock/Marion/ 
Wyandot Counties, OH 


Midwest 


Medium 


Marion 


1,150 


East Central Iowa 
(Cedar Rapids) 


Hidwest 


Small 


Cedar Rapids 


2,693 


Greater Omaha, HE 


Midwest 


Medium 


Omaha 


1,600 


Heartland, FL (Lakeland) 


South 


Large 


kwK€ 1 anu 




Jersey City, NJ 


Northeast 


Medium 


Jersey City 


1,600 


Larimer County, CO 
(Fort Collins) 


West 


Small 


Fort Collins 


1,200 


Macon/De Witt Counties, 
IL (Decatur) 


Midwest 


fmall 


Decatur 




Northeast, IN 
(Fort Wayne) 


Midwest 


Large 


Fort Wayne 


3,600 


Northwest, HN (Crookston 
and Thief River Falls) 


Midwest 


Small 


Thief River Falls 


550 


Oakland, OA 


West 


Medium 


Oakland 


1.065 


Providence/Cranston, RI 


Northeast 


Medium 


Providence 


1,750 


Springfield. MO 


Midwest 


Medium 


Springfield 


2,000 
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TABLE 5.2 

SDAs CONTACTO) AND PARTICIPATION RATE, 
BY PHASE OF SELECTION PROCESS 



*hase of Site 
Selection Process 



Total SDAs SDAs SDAs SDAs 

Contacted Participating Rejecting Dropped® 



Participation 
Rate (%} 



Date of Initial 
Contact ; 

Phase 1: Initial Design''/ 
Probabilistic'^ Selection 

(Before 1/6/87) 83*' 

Phase 2: Initial Design/ 
Expanded Recruitment* 

(Between 1/6/87 

and 4/30/87) 61 

Phase 3: Final Design''/ 
Expanded Recruitment 

(After 4/S0/87) es 



56 



22 



6.0 



49 



65 



13 



6.6 



8.2 



Date of Final 
Decision: 



ERIC 



Phase It Initial Design/ 
ProbabilSstic Selection 

(Before 1/6/87) 48 

Phase 2: Initial Des.ign/ 
Expanded Recruitment 

(Between 1/6/87 

and 4/30/87) 61 

Phase 3: Final Design/ 
Expanded Recruitment 

(After 4/30/87) 120 



34 



13 



2.1 



12 



47 



89 



11 



19 



4.9 



10.0 



Total 



229 



16 



170 



43 



7.0 



NOTES. SDAs were dropped because they were in the midst of administrative reorganization; 
they were facing a state takeover because of perfonnance problems; their program 
configuration could not be accommodated within the research design; or they were too 
geographically dispersed or served too few people. 
"Suimiarized in Table 4.4 

^The term is used because every SDA in a given category would have an equal 
probability of being selected into the sample. 

Seventy-three of these SDAs were contacted by MDRC under the probabilistic 
selection process. The other ten were not identified as priority SDAs under the 
probabilistic selection process, so no substantive discussions were held during Phase 1. In 
Phase 2. when the recruitment procedures were changed, MDRC recontacted most of these SDAs 

Under expanded recruitment MDRC was allowed to recruit sites under any given 
category without regard to the probabilistic selection process. 
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opponent could lead an otherwise wflling SDA to decline to participate.* A clear example of 
this occurred in Saa Antonio, Tcsas, where the SDA st?r sad PIC agreed to participate but 
their decision was overridden by a vote of the city council In other potential sites, the PIC 
and city officials supported participaUon but SDA staff opposed it. Second, it is likelj' that 
during the ihird phase some SDAs were aware that concerns had already been raised by others, 
and this "negative momentum" may have affected their own rftjpoosesJ 

A starting point for understanduig SDAs' reactions is a more detailed analysis of the 
concerns discussed in aapter 2. Table 53 shows is.sues raised by those SDAs agreeing to 
participate and those declining to, listed in the oi-der in which they were the most commonly 
dted by all SDAs. Both participating and non-participating SDAs raised similar issues. For 
example, 58 percent of participating SDAs expressed ethical and public relations concerns about 
how random assignment might be implemented in their program, whfle 62 percent of SDAs 
declining to participate raised tiiese issues. Overall, the most commonly cited concerns involved 
(1) potential controversies surrounding tiie study because of ethical and pubUc relations 
problems with random assignment or with tiie dem'al of services to controls, (2) the study's 
impact on meeting recruitment goals, and (3) Uie study's possible impact on performance 
standards. Eighty-eight percent of SDAs agreeing to participate in the study expreised at least 
one of tiiese interrelated concerns, a figure only slightly different from tiie 83 percent for SDAs 
not participating (not shown in Table 53). 

SDAs agreeing to participate were significantly more likely to raise a concern about 
possible effects on performance standards, potential grievances, and problems recruiting 
particular groups. Typically, these SDAs had more detaUed discussions with the research team, 
and they had to explore aU aspects of tiie study to develop detailed plans for its 
implementation. 

Local circumstances could largely determine tiie importance of tiie issues to each SDA 
and tiie willingness and ability of local staff to wrk with the research team to address them. 
Differences in local conditions, in fact, is what distinguished participating and non-participatuig 
SDAs. These differences could include botii objective, easily measured characteristics of SD.^, 



Chapter 2, service providers under pcrfonnance-based contracts were often the most 
difficult to con^ance of the merits of the study. 

Hie study bad been discussed in some JTPA newsletters and at national conferences of JTPA 
administrators. 
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TABUP 5.3 

PERCENT OF SDAs CITING CONCERNS ABOUT THE STUDY, 
BY PARTICIPATING AND NON-PARTICIPATING SDAs 



Percent of SDAs Citing the Concern 



Concern 



SDAs Participating SDAs Not Participating 



Ethical and Public 
Relations Implications of: 

Random Assignment 

in Social Programs 

Denial of Services to 
Controls 

Potential Negative Effect of 
Creation of a Control Group 
on Achievanent of Client 
Recruitment Goals 

Potential Negative Impact on 
Performance Standards 

Implementation of the Study 
When Service Providers Do 
Intake 

Objections of Service 
Providers to the '.Study 

Potential Staff Administrative 
Burden 

Possible Lack of Support by 
Elected Officials 

Legality of Rondom Assignn-^nt 
and Pbssible Grievances 

Procedures for Providing 
Controls with Referrals 
to Other Services 

Special Recruitment Problems 
for Out-of-School Youth 

Sample Size 



56.3 
50.0 



62.5 
68.8 

18.8 
12.5 
18.8 
6.3 
37.5 

25.0 

31.3 
16 



62.3 



54.7 



46.7 



22.2 



21.2 



17.9 



16.0 



16.5 



12.7 



13.2 



9.0 



212 



Total 



61.8 



54.4 



47.8 



25.4*** 



21.1 



17.5 



16.2 



15.8 



14.5** 



14.0 



10.5** 



228 



(continued) 
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TABLE 5.3 (continued) 



SOURCE: Based on responses of 228 SDAs contacted about possible participation in the National 
JTPA Study. 

NOTES: Concerns noted by fewer than 5 percent of SDAs are not listed. 

Percents may add to more than 100.0 because SDAs could raise more than one concern. 

A two-tailed t-test was applied to differences between participating and non- 
participating SDAs. Statistical significance levels are indicated as: * « 10 percent; « 5 
percent; and « 1 percent. 
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such as thoa reported in the JIPA Annual Status Report (JASR), and less clear^t. but stfll 
important, diflfereaces in leadership style or desire to be involved in the national policy debate. 
As the rest of this analysis shows, there are few clear patterns of differences on the 
char^ristics included in the JASR data, and an SDA's decision apparently was also affected 
by such intangibles as the personality of the SDA director or PIC chairperson and local 
poUtical circumstances in the SDA. 

^ A CoiDpari.son of PartieipaHnp and Non.Partidp »Hny CT >As Based nn j Agp n^.. 
Table 5.4 presents characteristics of partidpating and non-partidpating SDAs in program 
year 1986. when MDRC began contacting SDAs.« Partidpating and non-participating SDAs 
show statistically significant difference, on onl>- three of these characteristics: regional 
duitribution. population density, and number of program terminees in program year 1986.' 
SDAs agreeing to partidpate were more likely to be from the Midwest than were other SDAs 
contacted about the study. Seven SDAs ir. the study (44 percent) were located in the Midwest, 
as opposed to oniy 17 percent of non-partidpating SDAs. Partidpating SDAs also served a 
smaUer average number of people under Title HA tiian did non-partidpating SDAs.^^ in 
interpreting tiiis. it is important to realize Uiat smaUer SDAs are not necessarily more rural- 
among SDAs in the study, two im^er city areas (Oakland and Jersey City) served fewer people 
under Titie HA in program year 1986 than did tiie national average for aU SDAs. 

Table 55 presents a comparison of adult and youtii perfonnance measures for 
partidpating and non-partidpating SDAs. For each of the seven performance measures in 



prSlnTTmSfdi?J-S averages arc mentioned in theVext Appendix Se C J 

fact t^fdSrSn" thfr^^u^ '''' """^^^ '° ^ ^ in 

pool of potential SDAs for the study was identified using progitm vSm^S- tj^ 
500 termmees in that year were eUminated from the listf SoS SdS wTiJ^rl' Ih^^V^Z^ 
program year 1984 had fewer than 500 in program year 19^ ^ 
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TABLE 5.4 

SELECTED CHARACTERISTICS OF SDAs CONTACTED 
DURING SITE RECRUITMENT. BY PARTICIPATION STATUS 



Characteristic 



LOCAL SEHINfi 

Region (%} 
Northeast 
South 
Hidwest 
West 

Average SDA Unemployment Rate 
(% in program year 1986) 

Average Change in SDA Unemployment Rate 
from 1985 to 1985 (percentage points) 

Average Population Density 
(persons per square mile) 

SIZE OF PROGRAM IN PROGRAM YEAR 1986 

Average Teminees from Title IIA 

CHARACTERISTICS OF PEOPLE SERVED UNDER 
TITLE IIA IN PROGRAM YEAR 1986 

Average Percent of Teminees 
Who Are White 

Average Percent of Teminees 
Who Are High School Dropouts 

Average Percent of Teminees 
Who Are Welfare Recipients 

LENGTH OF PROGRAM SERVICES UNDER 
TITLE IIA IN PROGRAM YEAR 1986 

Average Weeks of Enrollment for Adults 

Average Wefks of Enrollment for Youth 

Sample Size 



Non- 

Participating Participating 
SDAs SDAs 



12.5 
25.0 
43.8 
18.8 



7.8 



0.1 



1.245 



957 



59.8 



25.5 



26.9 



18.3 
19.9 
16 



22.1 
30.0 
16.9 
31.0 



7.4 



-0.3 



1,172 



1,957 



54.9 



24.7 



28.8 



17.2 
17.9 
213 



Total 



21.4 
29.7 
18.8 
30.1 



7.4 



-0.3 



1.177** 



1.888 



55.3 



24.8 



28.8 



17.3 
18.1 
229 



(continued) 
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TABLE 5.4 (continued) 
SOURCE: JTPA Annual Status Report (JASR) for prografii year 19S6. 

<• sftt^Sif.w'if ™"' °^ "» «"«renMS fn individual ch.r.cterfs«cs 



TABLE 5.5 



SELECTED PERFORMANCE MEASURES OF SDAs CONTACTED 
DURING SITE REC«?UITHENT. BY PARTICIPATION STATUS 



Characteristic 
ADULTS 

Entered Baployment Rate 
Average 

Adjusted'' Average 

Wei Tare Entered Employment Rate 
Average 

Adjusted Average 

Wage at Placement 
Average 

Adjusted Average 

Cost per Entered Employment ($)^ 
Average 

Adjusted Average 
YOUTH 

Positive Termination Rate (%)^ 
Average 

Adjusted Average 

Entered Employment Rate (%)9 
Average 

Adjusted Average 

Cost per Positive Termination (%)^ 
Average 

Adjusted Average 



Participating 
SDAs 



77.7 
14.3 



71.0 
18.8 



5.04 
0.36 



2.838 
-1.595 



85.1 
10.4 



52.4 
11.3 



2.392 
-1.474 



Sample Size 



16 



Non- 
Participating 
SDAs 



72.4 
10.4 



64.0 
11.9 



5.18 
0.46 



2.829 
-1.615 



81.3 
7.5 



54.2 
13.5 



2.348 
-1.513 



213 



SOURCE: 



JTPA Annual Status Report (JASR) for program year 1986. 



Total 



72.5** 
10.7* 



64.5** 
12.4** 



5.17 
0.45 



2.830 
-1.613 



81.4* 
7.7 



54.0 
13.3 



2.351 
-1,510 



229 



Jo'r!nect'Jot'l'?lIrr'! T'"!'' '■"''■''■'""^ S^^^"'^" «nd -re intended 

to reflect local labor market conditions and characteristics of persons served. 

15 to 16 fo^nlrHrtr;- '^"cni ^^^^"^^^^ ^o-- individual perfomance measures varies from 
15 to 16 for participating SDAs and from 209 to 213 for non-participating SDAs. 

SDAS Statl'sti'cal'L-rnffT^'l 'Y'^^'''^''^ between participating and non-participating 
- J percent '^'S"'^'""" '''^'^ «re indicated as: • . lo percent; *. - 5 percent; 1 
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TABLE 5.5 (continued) 



The percent of all adult terminees who entered employment. 

Adjusted adult entered etnployment rate is the SDA's actual entered employment rate 
minus its performance standard. Other adjusted measures of perforaance are calculated similarly. 

The percent of all adult welfare recipient tenninees who entered employsnent. 

Average hourly wage of adults who entered efflployaent at termination. 
T-otal expenditures for adults divided by the number of adults who entered employment. 

The percent of all youth tcraiinaes who had a positive termination: either entered 
employment or met one of the employability enhancement definitions. 

SThe percent of all youth teminees who entered employment. 
. ■ expenditures for youth divided by the number of vouth who had a positive 

termination. (See footnote f.) «.■»!: 
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place in program year 1986, it lists the average performance of partidpating and non- 
participating SDAs and the average difference between actual SDA performance and the SDA's 
performance standard for that measure. For example, the top entry shows that the 
participating SDAs averaged a 78 percent adult entered employment rate, as compared to a 
73 percent average rate for non-participating SDAs. The second entry shows that parU'dpating 
SDAs on average exceeded their adult entered employment rate standard by 14 percentage 
points, as compared to 10 percentage pomts for non-partidpating SDAs. Several condusions 
can be drawn from Table 5.5: 

• Adult performance measures: Participating SDAs, on average, showed 
higher measured performance than did non-partidp-iting SDAs on two 
measures (entered employment rate and welfare recipient entered 
employment rate), while on the other two measures (wage at placement 
and cost per entered employment), the two groups of SDAs were 
similar. 

• Youth performance measures: Participating SDAs, on average, showed 
higher measured performance on the positive termination rate than did 
non-participating SDAs. The performance of the two groups of SDAs 
on the remaining two measures was generally similar. 

A further statistical analysis suggests ways in which these differences in characteristics 
might have affected the probability an SDA would agree to participate. The model of the 
decision process being estimated assumes that the probability an SDA contacted about the 
study would agree to partidpate was a function of the variables listed in Tables 5.4 and 5.5." 
The statistical analysis (reported in Appendix Table C3) finds, first, that there does seem to 
be a pattern of overall differences in observed characteristics between SDAs agreeing to 
participate and those not agreeing to participate.^^ Second, two variables seem to be 
important explanatory variables in an SDA's decision on participation: 



Tne detailed results of the analysis are presented in Appendix Table C3. The model used ordinai>' 
least squares regression techniques with an SDA's decision on whether or not to panicipate as the 
dependent variable and the SDA charaaeristics in Tables 5.4 and 5.5 as the independent variables. This 
technique was used to allow for a straightforward calculation of the F test for the regression. A model 
using logit techniques, not reported here, finds similar relations between the independent vanables and the 
partic^ation decision. 

' More precisely, the *p value* of the F statistic indicates that there Is only a 21 percent probability 
of obtaining the coemdent estimates reported in that table if the true chance of an SDA agreeing to 
partidpate did not vary with any of the measured characteristics included in the estimated model. The 
closer the 'p value* Is to one, the less of a difference there is in the average charaaeristics of participating 
and non-panicipating SDAs. 

ERIC 



• Region: LocaUon in the Midwest increased the probabiUty of 
partiapation. This could have been a measure of the number of 
agenaes mvojved in the operation of the SDA's program, since 
Midwestern SDAs tend to be more centralized than thcSTS other 
regions of the country. The fewer agencies involved, the less controversy 
and delay m arnvmg at a decision to partidpate. 

• Labor market conditions: A recent drop in the unemployment rate 
decreased the probabffity an SDA would agree to partidpate. TTus was 
closely hnked to recruitment difGcuIties in the SDA 

The coeffidents estimated for remaining factors included in the model were not statistically 
significant under the usual statistical tests, but most had the expected sign." 

Tlus analysis suggests some tentative lessons for recruiting sites in future studies where 
partiapation is voluntary. First, many factors that might have affected an SDA's dedsion do 
not seem to have been strong influences; many variables in the model did not have statisticaUy 
significant coefficients. Second, the number of organizations involved in making the dedsion 
about participating is important: The larger the number that must agree, the less likely is 
participation. n.is means that to get sites with a diversity of organizational stmcture. 
researchers must be prepared to devote more time and resources to discussing the study with 
sites in which many organizations are involved in program operations. Third, recent changes 
in the local environment - such as changes in the unemployment rate - that wiU affect the 
flow of clients into a program can be important, since the local agency may not have had time 
to adjust program offerings or client recniiting techniques to the new circumstances. This 
means that to achieve diversity on this characteristic - in this example, a mix of SDAs that 
have and have not had a recent change in the unemployment rate - researchers may have to 
provide some sites with technical assistance on client recniiting. 

^' Other Factors Possibly AfTprfin p an SDA's nerision on Partirip ^tinn 

Based on its contacts with SDAs. the research team believed that two other factors were 
important in some SDAs' decision to participate. These were not included in the previous 
discussion because they were somewhat subjective and difficult to measure. One was a 



,h. nr JS, r*" ^ ""'"S techniques, ihe number of Title IIA lerminees also affected 

t^i^ffi'^ '° S'"^" "ke'y to agree to paiSSpS T^fS 

also have been measunng the level of controversy sunounding crucial decisions in the IdI ^ 
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willingness by senior SDA staff to undertake innovative or risky projects, possibly becausie the 
SDA's administration was stable and the staff had solid political support in the community. 
The second was a site's desire to assure inclusion in the study of its typ& of SDA because the 
study's findings would affect naUonal policy. One rural SDA in a rarely studied part of the 
country cited this as a factor in its decision to participate. 

n. A Compariso n of SDAs in the Study to the National TTPA System 

The SDAs in the study were recruited from among SDAs in the forty-eight contiguous 
states with at least 500 or more persons terminated from their program in program year 1984, 
the last program year for which data were available when SDA selection began.^* This section 
discusses how the study sites compare to this pool of potential SDAs based on data for 
program year 1986. 

Table 5.6 presents data on SDAs in the study and on all SDAs in the pool from which 
these participating SDAs were recruited.^^ The data show that the two groups of programs 
are very similar. As in Table 5.4, these data are organized into four categories: 

• Local setting: SDAs located in the Midwest are overrepresented in the 
study, while those in the South are underrepresented; but the 
proportions from the Northeast and West are very similar to those in the 
pool of potential SDAs. Labor market conditions in participating SDAs 
were similar to those in the pool of potential SDAs, but participating 
SDAs tend to be more densely populated, though the difference is not 
statistically significant^^ While the participating SDAs do not include 
a large metropoUtan area, they do include inner city sections of the New 
York (Jersey City) and San Francisco (City of Oakland) regions. 

• Size of program: Participating SDAs on average served fewer people 
under Title HA than did the pool of potential SDAs. though the 
difference was not statistically significant 

• Cfaaracteristics of people served: The average percentages of terminees 



^''This process is also described m Chapter 4 and in Abt Assodates. ICF/NORC, and Manpower 
Demonstration Research Corporation, 1986. 

Appendix Table C4 shows more detailed unweighted comparisons of participating SDAs and the 
pool of potential SDAs for the study, and Appendix Table C5 shows similar comparisons w»th SDAs 
weighted by the number of Title IIA terminees in program year 1986. 

**The higher population density for participating SDAs is because three SDAs in the study (Jersey 
City. New Jersey; Providence/Cranston. Rhode Island; and Oakland, aiifomia) have quite high populauon 
densities. r r 
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TABLE 5.6 



SELECTED CHARACTERISTICS OF PARTICIPATING SDAs AND 
THE POOL OF POTENTIAL SDAs FOR THE STUDY 



Characteristic 



Participating 
SDAs 



Hon- Pool of 

Participating Potential 
SDAs SDAs 



LOCAL SEHINS 

Region (%) 
Northeast 
South 
Hidwest 
West 

Average SDA Unemployment Rate 
(% in program year 1986) 

Average Change in SDA Unemployment 

Rate from 1985 to 1985 (percentage points) 

Averaga Population Density 
(persons per square mile) 

SIZE OF PROGRAM IN PROGRAM YEAR 1986 

Average Terminees from Title IIA 

CHARACTERISTICS OF PEOPLE SERVED 
UNDER TITLE IIA IN PROGRAM YEAR 1986 

Average Percent of Terminees 
Who Are White 

Average Percent of Terminees 
Who Are High School Dropouts 

Average Percent of Tenninees 
Who Are Welfare Recipients 

LENGTH OF PROGRAM SERVICES UNDER 
TITLE IIA IN PROGRAM YEAR 1986 

Average Weeks of Enrollment for Adults 

Average Weeks of E nrollment for Youth 

Sample Size 



12.5 
25.0 
43.8 
18.8 



7.8 



0.1 



1.245 



957 



59.8 



25.5 



26.9 



18.3 
19.9 
16 



17.2 
36.5 
29.3 
17.0 



8.0 



-0.1 



772 



1.561 



59.9 



24.7 



28.8 



18.8 
19.4 
488 



17.1 
36.1 
29.8 
17.1 



8.0 



-0.1 



787 



1.542 



59.9 



24.8 



28.8 



18.8" 
19.4 
504 



(continued) 
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TABLE 5.6 (continued) 



from 



SOURCE: JTPA Annual Status Report (JASR) for program year 1986. 

rj^P'ic "'"^"^ ^o*" individual SDA characteristics varies 

15 to 16 for participating SDAs and from 487 to 504 for the pool of potential SDAs 

notenti«l JSi/'^fnt ^T^'"'? "'■^^"''^""s between participating SDAs and the. pool of 
potential SDAs (not including those th:t participated in the study). 

«rnnf < """"•''^ the di f f erences in individual characteristics are statistically significant 
except for regional distribution, for which statistics could not be calculated. ^"'"""^ 
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who were white, high school dropouts, and welfare recipients in the two 
groups of SDAs were virtually identical 

• Length of services: The two groups of SDAs enroUed adults and youths 
for about equal average lengths of JTPA participaUon. 

Table 5.7 presents information on performance measures for both groups showing that: 

• Adult performaace measures: Participating SDAs had higher entered 
employment rates (overall and for welfare recipients), but the two groups 
were very similar on other adult performance measures. 

• Youth performance measures: Participating SDAs had a higher positive 
termination rate than did other SDAs, but the two remaining youth 
measures were quite close. 

At this time, it is not possible to report how these differences in the two groups of 
SDAs might affect the impacts in the study sites. If the JTPA performance standards were 
correlated witii program impacts, the better-than-average performance of tiie study SDAs on 
three of the seven measures would suggest that Uiese SDAs would also have better-than- 
average program impacts. However, there is much uncertainty about the relationship between 
the existing JTPA performance standards and program impacts. Research in other related 
contexts (employment programs for AFDC appUcanits and redpients) found an inverse 
relationship between program performance on outcome measures and program impacts, and 
simUar concerns have been raised about JTPA." UntU the impact analysis in this study is 
completed and comparisons can be made between program impacts and measured performance 
on the existing performance standards system, the issue remains an open one. 

A further statistical analysis, reported in Appendix Table C.6, lessens these concerns 
somewhat It considered the extent to which the measured characteristics discussed above 
affected the probabUity that an SDA in the pool of potential SDAs would become a 
participating SDA.'^ This analysis found a "p value" of the F statistic of .4469, indicating a 45 
percent probabUity of obtaining the coefQcient estimates reported in that table if the true 
chance of an SDA agreeing to participate did not vary with any of the measured characteristics 
included in the estimated model. In this estimated model, only the variables for the West and 



^Sec Friedlander. 1988, and Job Training Partnership Aa (JTPA) Advisory Committee. 1989 
vers«s?o'n-~tirsDM "'"^ " " " "P"""" ^'"^^ P^"-P^""S 
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TABLE 5.7 



SELECTED PERFORMANCE MEASURES OF PARTICIPATING SDAs 
AND THE POOL OF POTENTIAL SDAs FOR THE STUDY 



- Measure 



77.7 
14.3 



71.0 
18.8 



5.04 
0.36 



ADULTS 

Entered Dnployment Rate (%)* 
Average 

Adjusted** Average 

Welfare Entered Employment Rate (%)^ 
Average 

Adjusted Average 

Wage at Placement ($)^ 
Average 

Adjusted Average 

Cost per Entered Employment ($)* 
Average 

Adjusted Average 
YOUTH 

Positive Termination Rate (%)^ 
Average 

Adjusted Average 

Entered Employment Rate (%)9 
Average 

Adjusted Average 

Cost per Positive Termination ($)*^ 
Average 

Adjusted Average 



Non- 

Ptirticipating Participating Pool of 
SDAs SDAs Potential SDAs 



2.838 
-1,595 



85.1 
10.4 



52.4 
11.3 



2.392 
-1.474 



72.4 
10.5 



63.7 
11.9 



5.06 
0.41 



2.932 
-1.511 



81.3 
6.9 



52.4 
12.2 



2.382 
-1.448 



72. 5** 
10.6* 



63.9** 
12.1** 



5.06 
0.40 



2,929 
-1,514 



81.4* 
7.0 



52.4 
12.2 



2.382 
-1.449 



Sample size 



16 



488 



504 



SOURCE: JTPA Annual Status Report (JASR) for program year 1986. 

NOTES: State performance standards for individual SDAs are set by governors and are intended 
to reflect local labor market conditions and characteristics of persons served. 

Due to missing data, the sample size for individual performance measures varies from 
15 to 16 for participating SDAs and from 487 to 504 for the pool of potential SDAs. 

An F-test was applied to differences between participating SDAs and tne pool of 
potential SDAs (not including those that participated in the study). Statistical significance 
levels are indicated as: * « 10 percent; ** « 5 percent; and « i percent. 
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TABLE 5.7 (continued) 



The percent of all adult terminees who entered employment. 

Adjusted adult entered employment rate H the SDA's actual adult entered employment 
rate minus its performance standard, other adjusted measures of perfonnance are calculated 
Similarly* 

jThe percent of all aduu welfare recipient teminees who entered employment. 
^Average hourly wage of adults who entered employment at termination. 
^Total expenditures for adults divided by the number of Adults who entered employment. 
The percent of all youth teminees who had a positive termination: either entered 
onployment or met one of the empl oyabi 1 i ty enhancement definitions. 

jThe percent of all youth teminees who entered employment. 

* . ®'^P*"''^*"'"« for youth divided by the number of youth who had a positive 

termination. (See footnote f.) 
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Midwest were statistically significant 

In summaiy, ^iiiile the SDAs in the study are not strictly representative of the pool of 
potential SDAs, they are similar on many dimensions and clearly exemplify the diversity of 
the JTPA system. Within the study SDAs, there is variety in participant characteristics, local 
labor market conditions, and measured performance. There is also variety in administrative 
structure and program services, as discussed in more detail in Appendix B.^^ As a result, the 
impact estimates will include findings for a range of naUonally fiinded employment and training 
programs. By subdividing the sample appropriately, it will be possible to examine how impact 
findings vary by type of activity, type of participant, and type of local environment (rural versus 
urban, among the regions, unemployment level, etc.).^ 

ni. The Sample Size for the National .TTPA Study 

The SDAs participating in the study produced a total sample of 20,606, as shown in 
Table 5.8. This sample is more than twice as large as the sample in any previous random 
assignment field study of employment and training programs except for the evaluation of thet 
Demonstration of State Work/Welfare Initiatives.^^ The characteristics of the sample members 
are mere varied than in previous random assignment studies, several of which focused on a 
more narrowly defined target group such as welfare recipients or dislocated workere. 

The sample includes large numbers of adults and out-of-school youth in all treatment 
groups. Percentages of adults and out-of-school youth in the sample are similar to those for 
JTTA terminees nationally.^ As is the case nationally for JTPA, a small proportion of 
minority out-of-school youth were recommended for OJT, but otherwise the sample sizes for 
white and minority out-of-school youth are over 850 in each treatment category. 



Appendix B provides a profile of each SDA jn t^e siudy, including infonnaiion on its administrative 
structure and program service emphasis. 

future analysis will compare the background charaaeristics and post-program earnings of those 
randomly assigned to the treatment group with a nationally representative sample of JTPA adult and out- 
of-school participants serx-cd during the period of random assignment The comparison sample will be 
drawn from the Job Training Quarterly Survey (JTQS). Information on post-program earnings for the 
nationally representative comparison sample and for the experimental sample will be obtained from 
government administrative records, allowing a comparison of program outcomes using identical data. 
^Sec the discussion of other projects in Chapter 1 and the summary in Table 1.1 
^BaseA on data for program year 1987 from JTQS. It will not be possible to compare service receipt 
for the study sample and JTPA terminees nationally until later in the post-random assignment follow-up 
period, when information on program participation is analyzed. 
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TABLE 5.8 

FINAL SAMPLE, BY TARGET GROUP AND TREATMENT CATEGORY 



Out-of-School Youth 
White 
Minority 



Treatment Category 



OJT 


CT-OS 


Other Services 


Total 


3,190 


1,592 


2,078 


6,860 


2,672 


3,417 


1.980 


8,069 


1,571 
1,027 

544 


2,097 
981 
1,116 


2,009 
873 
1,136 


5,677 
2,881 
2,796 



Total 



7,433 



7,106 



6,067 



20,606 
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Despite the large size of the final sample, it is 9^94 people below the original sample 
ceiling of 30,000 and 7,817 below the original target amount for the participating SDAs, as 
shown in Table 5.9. In most of the SDAs, the sample was greater than or near the target 
number. However, in three participating SDAs with large target samples, serious problems 
arose that led to major shortfalls; a fourth SDA (Fresno County, California) initiaUy agreed 
to participate and then withdrew from the study. The problems in these large SDAs illustrate 
common implementation challenges encountered in this type of research: 

• East Central Iowa: During the period of random assignment, this SDA 
experienced a large drop in the amount of funding available for new 
enrollments, for reasons unrelated to the study.^^ In program year 1987 
and previous years, the SDA relied on a surplus of funds carried over 
from past years and on large incentive awards to support a level of new 
enrollees that could not otherwise have been sustained. These surpluses 
were finally used up in program year 1987, and incentive avrards 
declined.^ This experience illustrates the difBculty of estimating sample 
sizes because of the need for accurate projections of new enrolhnents, 
not just total participation levels. 

• Heartland, Florida: Client recruitment problems increased in severity 
as the unemployment rate dropped, soon after random assignment began. 
Despite extensive technical assistance on cUent recruitment and retention, 
the SDA could not adopt new procedures quickly enough to avoid 
^laderenroUment problems, and it decided to end random assignment long 
»*-efore the target sample was reached- This illustrates the complete 
barrier posed to random assignment by serious cUent recruiting problems. 



• Springfield, Missouri: In this SDA, implementation of study procedures 
creat«J a burden on staff and the program. Over time, this diverted 
attention from other program responsibilities, including growing problems 
of recruitment in some areas of the SDA. Staffs anxiety about 
informing applicants of the results of random assignment during 
individual interviews contributed to the burden and led the SDA to 
request a shortening of the period of random assignment^ This 
illustrates the importance of understanding the emotional implications of 
various study procedures for the staff involved. 



23 

Only new applicants lo ihe program were subject to random assignment Individuals already enrolled 
in ihe program when the study started were not randomly assigned unless they left the program and laier 
sought to enroll again. 

^e severity of the drop in new enrollees was increased by the reluctance of SDA staff to shift their 
^Jviarecoramendations to lower-cost, shoner-term services such as job search assistance. 

^TTie research team encouraged SDA and service provider staff to inform members of the control 
group of their status by phone or through a letter to lessen this stress. 
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TABLE 5.9 

RESEARCH SAMPLE FOR THE NATIONAL JTPA STUDY, BY SDA 





Target Sample 


Actual Sample 


Difference 


-Capual Area, MS {Jackson} 


1,220 


1,478 


+258 


Concentrated Employment Program, 


825 


683 


-142 


toosa vaiiey, ga (Rome; 


1,800 


1,840» 


♦40 


uurpu^ vnri s bi /nueces County f TX 


1,500 


1,609» 


+109 


urawTora/ nancocK/Harl on/ 
Wyandot Counties, OH 


1,150 


1,154 


+4 


East Central Iowa (Cedar Rapids) 


2,963 


498 


-2,465 


Greater Omaha, NE 


1,600 


1,362 


-238 


nearLiona, rL (LaKeianCiy 


4,850 


597 


-4,253 


jersey uity, no 


1,600 


1,686* 


+86 


Larimer County, CO (Fort Collins) 


1,200 


1,027 


-173 


Hacon/Oe Witt Counties, 


750 


471 


-279 


norLneosi, In ^rOrt Waj^ey 


3,600 


3,608 


♦8 


Nortnwest, mk (Crookston and 
Thief River Falls) 


S50 


560 


+10 


Oakland, CA 


1,065 


1,072» 


+7 


Providence/Cranston, RI 


1,750 


1,759" 


+9 


Springfield, HO 


2,000 


1,202 


-798 


Total 


28,423 


20,606 


-7,817 


NOTE: "Some persoijs at this site were randomly essigned 
a ratio higher than 2:1. 


to the treatment 


or control group at 
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• FresBO Comity, CaUfoniia: Service provider opposition to the study, 
present during the early negotiations about the SDA's participation, 
reappeared after random assignment began and eventually forced tiie 
SDA to reverse its decision and withdraw from the study.^ Among the 
crucial concerns were persistent recruiting problems, leading sendee 
providers to fall short of enrollment goals, and negative press coverage 
of the project, which focused on individual hardship cases, Th?s 
illustrates the risk of including sites in a random assignment study when 
major groups in the community remain opposed to the project 

Setbacks notwithstanding, the National JTPA Study - because of its large sample size 
and tiie diversity of its sites - presents a um'que opportunity to study the impacts of 
employment and training programs. Whfle the sites are not a random sample of all SDAs, 
they are broadly representative of tiie variety of tiie JTPA system. Thus, much can potentially 
be learned from tiie impact results in tiiese study SDAs about how JTPA operates in a variety 
of different settings and for different types of cUents. 



More ihan 700 persons had already been randomly assigned when ihe SDA made its final decision 
to withdraw from the study. Because the SDA decided it could no longer comply with the rule against 
serving controls in JTPA for 18 months after random assignmeai, none of the sample was usable m the 
analysis. 
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CHAPTER 6 



LESSONS FOR F UTURE RRgPARCH FROM THF. 
IMPLEMEOTATTON O F THE NATIONAL .TTPA STUnV 

Implementation of the National JTPA Study is moving random assignment field research 
to new levels. Its sample of more than 20,000 individuals in 16 sites is the largest experimental 
sample ever assembled in an ongoing, voluntary employment and training program. The 
estimation of treatment-specific impacts requires an unusually complex research design. In the 
process of implementing the study, researchers have confronted famiUar issues in more intricate 
forms and new issues not seen in past research. 

The research design implemented in the sites wiU achieve most of the study's original goals. 
It wiU provide impact estimates for the overaU JIPA program in participating sites and 
estimates for specific parts of their program. These estimates will be made with relatively 
minor changes in the program. However, some issues (such as recruiting a representative 
sample of sites) proved intractable, requiring modification of the study's original goals and 
representing basic tradeoffs in doing this type of research. Others have stimulated the 
development of promising new approaches. 

WhUe the National JTPA Study is a one-time effort - designed to estimate the program's 
impacts, improve methods of monitoring and studying the system, and provide a comprehensive 
data base for further research on JTPA - the early experience of implementing the study can 
provide valuable lessons for other research efforts as weU. The Department of Labor itself 
may thus benefit more broadly. Having acknowledged the considerable difficulties of 
«onexperimental research and, in the JTPA study, having made a commitment to greater 
reliance on experimental methods, the department continues to se^k better ways to study the 
ongoing programs under its direction. Furthermore, other agencies - at the federal, state, and 
local levels - face similar problems in evaluating programs and can learn from the JTPA 
experience.^ 



Programs presenting some comparable issues include the Department of Agriculture's ongoing 
evaluation of the Food Stamp Employment and Training Program (Abt and Westat. 1988; Puma. 1988) 
and the Department of Health and Human Services' evaluation of the Job Opportunities and Basic Skills 
Training (JOBS) Program under the Family Support An of 1988 (U.S. Department of Health and Human 
Services, 1989). 
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This chapter provides an overview of the implementation I«sons to this point ty outlining 
the unique features that make the study chaUenging and by presenting several genera! principles 
that have emerged. The chapter then offers more specific lessons concerning the implement- 
ation of random assignment in a study of an ongoing program such as JTPA. Many of the 
issues it discusses would also arise in nonexperimental research, although often in a somewhat 
different form. 



An Overvie w of ihe Early Lessons from This Stndv 

Hie National JTPA Study breaks new ground in several ways, and its "newness" is one of 
the reasons implementation is so chaUenging. As an overall assessment of ongoing local JIPA 
programs, it moves experimental research squarely into ongoing program operations with few 
spedal inducements for local sites to partidpate. Its implementation requires SDAs to do 
things outside their normal experience. Hiey are asked to submit their entire program to a 
rigorous impact study, to exclude a portion of tiie volunteers applying for tiieu: program from 
all JIPA activiti'es for an extended period, and to comply witii procedures for selecting persons 
to enroll tiiat could affect thsir abiUty to meet performance standards. While similar types 
of research had been conducted in welfare employment and training programs and smaller 
demonstrations within JIPA, at tfie time tiie project began tiie research team co»iia not point 
to directly comparable successes in past research to assuage site concerns. 

JTPA's characteristics as an employment and training system also pose special challenges 
for random assignment research: 

^ JTPA authorizes and funds a broad list of activities, rather than a single 
program model Hence the diversity of local programs is great This would 
raise issues for any impact analysis, but it presents special problems in 
experimental research because a single random assignment design (and 
accompanying set of procedures) must be integrated into the normal operations 
of a great vanety of local programs. 

• JTPA is a voluntary program, and since local SDAs often face difficulties 
recruiting clients, creation of a no-JTPA services conirol group poses serious 
problems. 

• Important JTPA program decisions are made by many individuals, representing 
the diverse perspectives of p.nvate i.ndtistiy representatives, service providers, 
professional JTPA program administrators, and local elected ofilcials. This 
includes the decision about whether to partidpate in the study and makes the 
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site selection process more complicated than in other experimental studies of 
program impacts. 

• JTPA already has an extensive performance assessment system, which is used 
to identify agencies in need of state supervision and to dispense incentive funds 
to reward agencies performing well Thus, many SDAs are concerned about how 
participatuig in the study might affect their performance. 

• JTPA is funded almost entirely by non-local sources, giving local ofGcials less 
financial incentive to conduct a rigorous study of the program's benefits and 
costs than might be the case if they funded the program themseh^es. 

The study's multiple - and potentially conflicting - research goals represent an additional 
new chalJ«?nge for experimental research. Random assignment, properly implemented, can 
assure the internal validity of impact estimates for the sites in the study. But - as recognized 
from the beginning of the project - in a system Uke JTPA, the complex research design of the 
study, voluntary participation of sites, and lack of special program mon^ for sites participating 
make it difficult or impossible to randomly select sites for this research. 

The desire for impact estimates both for the program as a whole and for individual types 
_ of treatments in normal, undisrupted local JTPA programs also posed a tradeoff; in this case 
in the choice of the point at which random assignment would take place. In JTPA, some 
services (generally job search assistance) are often provided before staff develop a formal 
service plan and recommend services. As a result, no single point of random assignment would 
allow an estimate of the impact of all program services (including these preUmmary ones) and 
the impact of specific types of services recommended in the service plan. Furthermore, some 
approaches that would facilitate the implementation of random assignment - for example, 
greater outreach in recruitment - would represent changes in the program and thus be 
inconsistent with the goal of studying it as it normally operates. Balancing these research goals 
within a single study led to a complicated research design requiring extensive cooperation from 
participating sites. 

The early experience implementing the National JTPA Study suggests five general 
principles for future research: 

1. Researchers must identify tradeoffs in research goals quickly and work with 
agencies sponsoring the study to set priorities for developing the research 
design. 
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The department and the research team were aware from the beginmng of the National 
JIPA Study that some of its mulUple objectives could be in conflict TTie department 
consciously adopted this ambitious plan, which ejrtended random assignment field research well 
beyond its past uses. Much effort went into tiying to implement what turned out to be an 
over-coi strained plan. 

By the time the research team and the department identified the objectives to emphasize, 
time .'ind momentum had been lost Many SDAs that would have been good candidates for 
participation under the final research plan had already rejected the offer, and membere of the 
research team were never able to reverse an unfavorable decision. 

In the case of the NaUonal JTPA Study, when the department clarified its priorities for 
the study, it emphasized the goal of internally valid estimates of net impacts for the sites in the 
study (both for their programs as a whole and for specific types of treatments) over the goals 
of statisUcaUy representative sites and unchanged program operations. TTiese choices were 
needed to develop a detailed research plan. 

In other studies, the choice of priorities might be different: An agency might choose to 
scale back the impact questions in order to reduce the burden on sites and to increase the 
chances of enrolling representative sites. In any case, early identification of confiictmg goals 
and resolution of priorities will greatly aid the research effort 

2, Mandating participation of sites in a random assignment field study is not a 
simple solution to the difficulty of recruiting a representative sample of 
pit)grams. 

In some past nonexperimental studies of the impacts of employment and training programs, 
the department required a random sample of local agencies to participate. Attempting this in 
a random assignment field study, especially one of a voluntary program, would open a host of 
new issues that would be difficult to solve. 

In experimental impact research, site staff are responsible for the most difficult part of the 
effort They must clearly and positively explain to applicants the ground rules for entry into 
the program, follow random assignment procedures, comply with the results of random 
assignment in providing services to those in the experimental group, and assure that members 
of the control group are not served in the program. If these procedures are not followed 
properly, the entire research effort fails. 
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There arc two serious obstacles to mandating participation in random assignment field 
studies. First, many types of cooperaUon necessary for the project are difScult to specify in 
detail and hard to monitor. For example, the tone with which intake workers explain the 
ground rules can affect the response of appUcants and the resulting burden and stress on staff. 
In this situation, defining what constitutes compliance with the study procedures is hard, and 
determining if the required staff behavior is occurring requires expensive, on-site monitoring. 

Second, a federal agency has limited leverage over local programs in a decentralized grant- 
in-aid system such as JTPA. States are in much closer contact with local programs than is the 
federal agency, and states play the central regulatory role. Without expUcit statutory language, 
a federal agency may not have the authority to mandate participation in a random assignment 
study. Unlike nonexperimental research, random assignment field studies require changes in 
enrollment and service selection decisions ordinarily left up to the local programs. 
Furthermore, even if participation could be required as a condition of the grant, the only 
sanction for non-cooperation would be a cut in federal funding, a rarely used strategy. 

In the final analysis, the research team must depend on a site's commitment to complying 
with the study rules. In the JTPA study, the department and research team clearly explained 
the procedures and likely difficulties to SDAs wnsidering participating; those that chose to 
enter the study did so with a fuU understanding of their responsibilities and a real commitment 
to the undertaking. This lessened the need to commit large amounts of resources to 
monitoring the day-to-day implementation of study procedures. 

3. Sites must b€ provided with specific benefits from participating that exceed the 
cost of the efTort involved in implementing the study. 

Research on prograih impacts is, in economic terms, a "public good." The major benefits, 
in terms of increased knowledge about program impacts at the subgroup and treatment group 
level, will be primarily based on the entire sample and thus be relevant to all local programs 
in the system. Something else - specific benefits to participating sites - must be offered to 
induce participation. 

In eariier national demonstrations such as Supported Work, participation in the study 
brought substantial special program funding, allowing an increase in overall service levels. In 
recent years, most studies have not provided special program funding for agencies, so other 
types of compensation and benefits have been needed. In the case of studies of welfare 
programs, states have a substantial financial stake in research findings since they partly fund 
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public assistance beneBts. TTie research findings of positive impacts have led states to expand 
welfare employment programs and achieve substantial savings in benefits. In this setting, states 
have been willing to contribute to the costs of studies, but in general it has been necessary to 
compensate agencies for the extra administrative efifort involved in participating. 

These developments have increajed the need for nonfinancial incentives to participate, 
nie JTPA experience offers a variety of examples of how this can be done without providing 
funds beyond those needed to cover the added administrative effort arising from the study. 
Partidpating sites will receive spedal reports or briefings on their clients' characteristics and 
tiie impacts of their program, tiiough sample size limitations may prevent precise subgroup or 
treatment-specific impact findings in smaUer sites. When tiie findings become available, they 
will have an opportunity to discuss them with tiie research staft In addition, they receive 
information on program operations (such as participation rates); technical assistance on study 
methods, client recruitment, and data collection and processing; and a chance to participate in 
the national debate on employment and training policy. These types of benefits can be offered 
in many studies. 

V/itiiout a doubt, one of the most difficult problems faced in the National JTPA Study has 
been finding a way to provide benefits to site line staff who implement tiie study on a daily 
basis. Site managers and officials - seeking better information to plan their program and 
consdous of resource constraints that prevent service to all eligibles - can see tiie benefits of 
the study clearly. But much of the work of tiie research falls on individuals who may have a 
very different perspective: the SDA and service provider line stafi", whose mission is service 
provision and who must turn away people tiiey feel they could help. 

As a partial answer, the research team arranged special meetings (which a representative 
of the Department of Labor typically attended) to make clear to the line staff that aU 
concerned were aware of their centra! role in the study and appreciative of their cooperation 
and efforts. 

4. In a performance-driven system, it Is especially Important for the research 
team to have an In-depth understanding of program procedures in order to 
anticipate problems and to develop responses that preserve the cor« of the 
study. 

When program operators have financial and other incentives to achieve a high percent of 
favorable post-program outcomes or to enrol! certain numbers or types of clients, many of the 
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daily procedures of the administering agency and its service providers change in response. 
Recruiting, intake, and enroUment practices; service offerings; and contractual relations witii 
service providers all are likely to reflect tiiese incentives. Superimposing random assignment 
procedures on these existing arrangements will inevitably be problematic. 

The research team should include persons familiar with the operational practices and 
incentives of tiie system. The National JTPA Study team included experienced JTPA 
administrators, persons knowledgeable about JTPA data systems, and experts in evaluation 
methods. This combination increases the chances that the original research plan will avoid 
many operational problems and helps the team develop alternatives that address the needs of 
site staff without sacrificing central research objectives. It also greatly increases the credibility 
of the research effort with site staff; who do not have to explain their concerns to researchers 
who may appear unaware or unconcerned about them. 

5. Implementing a complex random assignment research design in an ongoing 
program providing a variety of services does inevitably change Its operation 
in some ways. Researchers should Identify the key aspects of a program to 
leave unchanged and seek to develop a research plan that retains them. 

The most likely difference arising from a random assignment field study of program impacts 
- especially in a voluntary program - is a change in the mix of clients served. Expanded 
recruitment efforts, needed to generate the control group, draw in additional applicants who 
are not identical to the people previously served. A second likely change is that the treatment 
categories may somewhat restrict program staffs flexibility to change service recommendations. 
It is also possible that the added administrative tasks involved in the study, if not funded to 
support the staff involved, could force programs to divert staff from what they would otherwise 
do, such as counseling, case management, or job development 

In the JTPA study, the department and research team sought to leave the services 
provided by participating SDAs as unchanged as possible and to assure that SDAs and service 
providers randomly assigned only applicants they found eligible and appropriate for JTPA 
However, the department and research team also recognized the need for expanded outreach 
and the changes this could cause in the client mix. Technical assistance was provided to aid 
sites in recruiting more applicants, and in some SDAs staff - possibly because of this expanded 
outreach - ended up serving a somewhat less job-ready group of clients during random 
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assignment than they usually served-^ TTie treatment categories used in the JTPA study, were 
defined in the least restrictive way possible vMs still allowing estimates of separate impacts 
for persons recommended for OJT and classroom training. Finally, the department sought to 
compensate SDAs for all the administratwe effort invoh^ed in the study to avoid diverting staff 
away from normal service deliveiy. 

In other studies of ongoing programs, researchers might make different choices about the 
aspects of the program to protect from change. But to achieve the overall goals of the 
research - to study the program, insofar as possible, as it normaUy operates - it is important 
to recognize the likelihood of change, identify changes that would cause the most serious 
analytical problems, and design the analysis plan to avoid them. 

n. Lessons on Specific Tasks in the Research 

A* Lessons on Site Selectio n. Developing Study Procedures, and Reaching an 
Agreement To Participate 

The experience of the National JTPA Study suggests six lessons for picking sites, 
-developing study procedures, and securing sites' agreement to participate: 

1. Statistical representativeness of sites in random assignment field H^search 
such as the National JTPA Stiidy is an elusive goal that presents dear 
tradeoITs with other research objectives. 

The experience of the Nationa' JTPA Study iUustrates the difficulty of reaching agreement 
with randomly selected sites, especially when they operate voluntary programs and are 
experiencing recruitment problems. While the percentage of sites agreeing to participate did 
increase over time as the research design was modified, the rejection rate remained over 90 
percent because of operational and other concerns of the sites and the adverse publicity 
surrounding the original research design- 
Four options could increase the chance, of including a statistically representative sample 
of sites in studies like JTPA, but each presents its ov/n advantages and disadvantages: 

a. Mandate participation. In pa« nonexperimental research and program reviv-'ws, 
federal agencies have required local programs to participate. However, as noted above, subtle 



It is difficult to determine the extent to which ihc study caused this, since the unemployment rate 
m most sites dropped sharply during the study. Applicants for employment and training programs m 
strong labor markets typically are less job-ready than those in weaker labor markets. 
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types of cooperation are needed on a daily basis and compliance with study procedures is 
impossible to mandate or monitor continuously. Furthermore, some SDAs, because of severe 
recruitment problems or up-front services, cannot implement the type of random assignment 
model needed to answer the various impact questions without major changes in procedures. 
Hius, it is possible that mandated participation could compromise the internal validity of the 
impact estimates. 

b. Select many SDAs and greaUy reduce the raUo of controls to experimentals. 
One of the most troubling aspects of the study from the SDA perspective is the obligation to 
deny services to controls. Dropping the control ratio greatly, possibly to only 10 percent or 
less, would increase the attractiveness of the research design to SDAs. However, the drop in 
the number of controls would eliminate the possibility of site-speciOc findings, one of the major 
benefits of participating.^ To counteract this, the department would either have to mandate 
participaUon (increasing the need for close monitoring of sites)* or increase site payments 
above the level of the administrative costs associated with participating in the study. 
Furthermore, overaU research administrative costs would rise sharply with the increase in the 
number of sites: Many of the costs of implementing and monitoring random assignment are 
fixed costs per site that do not change greatly with the number of people randomly assigned. 
Manuals must be prepared that reflect local procedures; forms must be developed; staff must 
be trained; and procedures must be developed and implemented to monitor compliance with 
study rules. For similar reasons, survey costs wiU also increase with more sites. Finally, if the 
number of sites was substantially higher, it would be much more expensive to provide a 
detailed picture of each individual program. But without this, interpretation of impact 
estimates would be much harder because impact and program implementation research could 
not then be integrated.^ 
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*rhe precision of impact estimates for the entire sample is a function of the total sample size and 
the fraction of the sample assigned to the control group. Moving the fraaion assigned to the control 
group away from one-half in either direaion lessens the precision of estimates. For example, if the control 
fraction IS reduced to one-fourth, the overall sample must be 1333 times the size needed when controls 
constitute one-half the sample for impact estimates to retain the same level of precision (see Cave, 1987) 
The precision of the impact estimates for the overall sample could remain acceptable if the overall sample 
were very large and a small fraction were assigned to the control group. However, site-level impact 
estimates would be veiy imprecise. 

^Monitoring costs would increase because participating sites were not volunteers. 

In many decentralized programs Ukc JTPA, the available site-level data on program operations are 
veiy limited. In JTPA, for example^ there is information on participant characteristics but very bitle on 

(continued...) 
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c Provide enhanced program funding (possibly after the stndy ends) to create a 
financial incentive to participate. If sufficient funds were avaUable, local programs could be 
induced to participate by being offered the opportum'ty to increase local services. The obvious 
constraint here is budgetary. Supported Work, the first national random assignment field study 
of employment and training services, provided major support for program services," but the 
current budgetary situation does not allow for such spending. In addition, sites selected 
randomly for participation could recognize tiieir "monopoly" status and negotiate for very high 
funding. Furthermore, if Uie funding were made available dut'^ig the period of tiie research, 
that would change tiie program, undermining the goal of studying the program as it normaUy 
operates. 

d. Simplify the research design to make participation less bunlensome. Simple 
random assignment designs are much less difficult for sites to implement Such designs provide 
only for early random assignment of a portion of program-eUgibles or appUcants to a control 
group.6 Shiffing to such a design could greatly increase the acceptance by sites. Such a 
change, however, would preclude answering tiie type of treatment-specific questions central to 
a project like the National JTPA Study because random assigmnent would occur before staff 
designate a type of service for each applicaat' Studies with early random assignment into tiie 
program or to a control group wiU leave unanswered many of the "inside the black box" ques- 
tions that are tiie most important and policy-relevant in employment and traming policy today. 

In sum, experience suggests tiiat when treatment-specific impact estimates are important 
for a program like JTPA. each of the preceding options has disadvantages, and the realistic 
goal in site selection is to enlist a group of localities that capture the diversity of tbe system 
on measurable characteristics. As sites are selected, a small number of key characteristics of 
sites that are thought to influence impacts should be monitored, and recniiting efforts should 
concentrate on sites that would provide the desired variety. Once impact results are available, 
researchers should examine whether there is consistency among the diverse sites (which 



^(...continued) 

the nature of program services. Thus, information on program struaure has to be collected through 
interviews and document review at the local level. wii«.icu mrougn 

^S!^ "L"""^'' li?^ ""^"y P^* P™j"^ including the Work/Welfare Demonstration, 

the Food Stamp Employment and Training Program Study, JOBSTART. and Supported Work. 

o... JZl '^"Jl r«»«archcis from identifying a control group for types of activities and thus rule 

out experimental impact estimates for individual types of activities. 
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strengthens the case for generalizability) or similar results among sites found to be similar 
(which may suggest how to generalize the results). 

2. Researchers must be prepared to address the veiy dffferent interests and 
concerns of the many parties involved in local pr.fgrams. 

Caiapter 2 describes the variety of agencies and individuals involved in local JTPA 
programs. JTPA is by no means unique in this regard. In times of serious budget constraints, 
few programs have the resources to provide all services in-house or even to control - through 
contracts - aU aspects of service provision. Many programs entail collaboration among 
agencies.^ 

While the concerns of each party in a local program are to some extent unique, there are 
patterns in their interests. Program managers and boards of directors (the Private Industry 
Councils - PICs - in the case of JTPA) tend to be those most interested in infoimation (such 
as estimates of program impacts) affecting decisions about resource allocation. They may also 
be quite concerned about the public image of the program. Agency managers and line staff 
are most concerned about the day-to-day administrative burden the study procedures may 
create and how those procedures will personally affect thsir clients. Service providers for the 
program who are operating under performance-based contracts are very concerned about study 
procedures they perceive as jeopardizing their ability to reach contract payment points. 

3. Time is needed before the start of the study to learn about the details of 
the service system, to develop random assignment procedures, to sell the 
line staff and service providers on the benefits of the study, to train staff, 
and, posslbiy, to expand recruitment efforts to generate the needed 
number of applicants. 

Though reaching general agreement with a site to participate is a milestone, much remains 
to be done before the study will prog: ess smoothly. Obvifjusly, researchers must develop study 
procedures that interfere no more than necessary with normal program operations. Less 
obviously, there are payofis to briefing line staff (or their representatives) and other affected 
agencies about the study before a final agreement is reached and to giving them an 
opportunity to voice concerns and ask questions. Training staff in study procedures is another 



In the work/welfare area. California's Greater Avenues for Independence (GAIN) program - the 
nation's best-funded education, employment, and training program - involves extens, cooperauon among 
agencies. Sec Riccio et al., 1989, for a discussion. The JOBS litle of the Family bu^pon Aa of 1988 
envisions a similar kind of collaboration among service agencies. 
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important step. Finally, in voluntaiy programs, recruiting practices cannot be changed 
immediately. Adequate time to make changes and begin -to see a payoff can smooth 
implementation of random assignment 

4. Sites' concerns that a study may affect measured performance under 
existing assessment systems must be addressed, but a guarantee that sites' 
taUngs will not change is neither practical nor - in many cases - 
appropriate. 

ParticipaUon in a random assignment study may not affect measured performance at all 
To the extent that random assignment leads agencies facing outcome-based standards to serve 
less job-ready appUcants, measured performance may decline.^ But if standards are adjusted 
to reflect client characteristics, as they are to some extent in JTPA, the agency's rating may be 
unaffected. Of course, many factors unrelated to the study could affect measured performance, 
and the agency should not be held harmless for its response to them. Changes in the loJ 
economy, in services provided, or in funding levels could have an effect. Fmally, relaxing aU 
performance standards (as a guarantee would do) wo 'Id mean that the program is not being 
studied as it normally operates. 

Sites considering participating in the National JIPA Study often wanted assurances that 
the incentive funds they received under the performance standards system would not be 
jeopardized by participation in the study. TTie department, research team, and most states that 
were approached about this felt that a guarantee of unchanged mcentive funds was 
inappropriate. This meant that arriving at an agreement about proper adjustments to make if 
the study did affect measured performance was one of the most complicated aspects of the 
entire JTPA study.^*^ 

S. To gain sites' agreement to participate in a study like JTPA, funding to 
cover the administrative effort Involved must compensate sites for all the 
costs of participation, Including the costs of extr« recruitment of 
applicants and of processing the applications of Individuals who do not 
become a part of the research sample. 



•n agency was unable to recruit a -surplus' of applicants with characteristics similar 

to those previoi^ly served. In this case, random assignment of a portion of all applicants to a control ^oup 
would fot-ce the agency to 'dig deeper' into the eligible population and enroll harder-to-scrve clienti 

TT^ issue was especially complex because of the desire to study JTPA as it normally operates. The 
issue also raised conams about cross-program equity when - as is the case in JTPA - programs within „ 
state compete for a fixed amount of incentive funds. ^ 
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As discussed earlier in this chapter, when sites do not stand to benefit financially fi-om the 
results of the research, it is usually necessary to compensate them for the costs of participation, 
which may include added recruitment and cUent intake. Many voluntary programs cannot rely 
solely on walk-in applicants and referrals; they must seek appUcants through extensive outreach 
efforts. If the study requires them to increase their applications, these costs must be covered. 
Furthermore, in most organizations, some appUcants do not proceed to the next step in the 
agency's intake process. To the extent that study procedures begin early in this process (before 
random assignment actually occurs), the study-related adramistrative effort will invohre more 
people than the actual research sample. In the case of JTPA, study forms were filled out in 
some SDAs at initial contact with the program. As a result, many more people filled out the 
form and were told about the study than were actually randomly assigned. Formulas developed 
to compensate sites for administrative costs must recognize this. 

6. The formal agreement to participate must give sites the option to 
terminate the agreement, with appropriate notice. 

In the National JTPA Study, sites did not lightly enter the project; the challenges it 
presents are obvious to all. Those who sign the agreement to participate are consequently 
committed to the effort However, unanticipated issues can arise, and forcing resistant sites 
to continue participating in a complex study raises many of the same issues raised by 
mandatory selection When sites see that researchers are willing to assume an obligation to 
address issues that may aiise, their anxiety about participating in an unfamiliar project declines. 
Giving sites the option to change their mind will lead more of them to say "yes" in the first 
place. 

B. Lessons on Client Recruitment 

1. In a voluntaiy program operating under performance standards or 
enrollment goals, it is essential to address client recruitment problems 
and assure sufficient applications for the program. If this is not done, 
the study can face serious ethical and operational problems. 

The greatest ethical concern about random assignment is that it potentially can reduce 
the number of people served because of the creation of a control group. However, if 
applications for the program exceed available slots by an amount at least equal to the size of 
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the control group - either through nonnal outreach or with special efforts during the study 
~ creation of a control group merely affects the choice of v^^iom to serve. To lake a 
hypothetical example, if the program has fimds to serve 100 people and at least 150 apply, 
assigning one-third of the appUcants to the control group does not reduce the total number 
served. 

However, if a program has serious recruitment problems, creation of the control group 
may mean that the service provider wiU be asked to turn away people ^e program slots 
remain unfflled. To continue the hypotheUcal example, if fewer than 150 persons apply for 
the program. creaUon of a control group consisting of one-third of the appUcants would reduce 
the number served to fewer than the 100 slots available. Furthermore, program operators with 
performance-based contracts would immediately feel the financial impIicaUons of under- 
emoUment and would request reUeL 

For these reasons, researchers must anUcipate potential recruitment problems, address 
them early, and continue throughout the study to assure that there are enough appUcants so 
that even with the creaUon of the control group sites are able to serve the maximmn number 
for whom funds are avaflable. As was done in the JTPA study, attenUon to this issue should 
begin as early as the site selecUon pha:^ Programs that have had recruitment problems in the 
past are poor candidates for a study unless a credible corrective action plan is put in place. 
Whfle this may lessen the representativeness of the sites in the study, ignoring the problem can 
put an abrupt halt to the entire project Techm'cal assistance on cUent recruiting can play an 
invaluable role in easing the administration of the project 

2. Many recruitment problems are the result of discretionary administrative 
practices and can be addressed if planning time is available. 

Most SDA and service provider staff are experts in employment and training services, not 
in markeUng their "product" In JIPA, for example, many SDAs promote their program 
using methods far from "state-of-the-art" - emphasizing the features of their program (e.g., 
funding sources, eligibility rules, program services, etc) rather than its beneGts for potential 
cUents (e.g., the improved Ufe to which the program can lead). Further, most of those who 
express an interest in JTPA programs never participate; in many SDAs. only 25 percent of 
those contacting the SDA about the program ever enroll. Providing sites with technical 
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assistance in client recruiting and retention can help address potential under-enrollment 
problems.^^ 

In three situations, however, recruitmg practices are quite hard to address. Erst, when 
the labor market is veiy strong and jobs are plentiful, agencies typically have a difficult time 
recruiting clients. Second, some agendes dehT>erateIy erect a time-consuming intake process 
with many hurdles to "screen clients for motivation." Whether these procedures accomplish 
their goal is debatable,^^ but 50^*^2 agencies resist changing them for fear of getting less 
motivated and capable applicants. Third, when recruitment is low because the service offerings 
do not match the needs of program-eligibles, only a major overhaul of the agency's service plan 
can solve the recruiting problems. 

Technical assistance in client recruiting and retention can have an added benefit for the 
research: It can serve as one of the major inducements to participate in the study, a benefit 
with payoffs lasting long after the study is over. 

C. Lessons on Developing Random Assignment Models 

The National JTPA Study suggests three lessons on de. .loping random assignment models 
to calculate net impact estimates for specific treatments ati for the overall program: 

1. The choice cf the point of random assignment is likely to involve a 
balancing of research goals. 

In an ongomg, voluntary program like JTPA, it is common for staff to provide some 
services (usually job search assistance) early in the intake process, before completing the 
service plan, which designates the major program service(s) recommended for the applicant^^ 
If staff do provide services this early, there will be no point of random assignment that will 
both precede the delivery of any service and follow the creation of a service plan. Therefore, 
it is impossible, without changing the program (Le., eliminating the up-front services), to 
estimate both the impact of all program services and the impact of specific types of services. 



In ihe JTPAsiudy, recniiling experts prepared several guides for staff and provided in-person irainmg. 
See Kelly, 1987; Elsman, 1987. 
^^See Kelly, 1987. 

^^is is in keeping wiih the besi recruiiing and reienUon praaices. Recruiting experts lypicallj advise 
staff to emphasize the benefits of panidpaiing (and to provide services earl>) before clients must pa> the 
"costs' of panidpaiing (in the form of paperwork, testing, and documenting their eligibility). See Kell>, 
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If the point of random assignment is placed early to capture aU program services,, it is 
impossible to estimate impacts for specific kinds of services. If it is placed after 
recommendaUon of a service plan, some program services are missed. 

One option when estimating the impacts of specific services in a voluntary program is to 
give staff as much time as possible for assessment and recommendation of appropriate services 
by delaying random assignment until just before services are to begin. Hiis delay also raises 
participation levels in the selected services because those chosen for the treatment group can 
quickly start their programs; the increased participation level, in turn, increases the statistical 
precision of the impact estimates. However, tiiis choice puts a greater burden on the sites 
because they have to cany more individuals Uirough their intake process to assessment 
Furthermore, it increases the emotional burden on staff and cUents, since there is more 
involvement with the eventual controls before they are informed of their status. 

2. It can be difficult to define categories of treatments for staff to choose 
among before random assignment that are narrow enough to be 
meaningful but consistent with normal practices. 

Most JTPA participants end up receiving only one of the more intensive services such as 
OJT or classroom traimng in occupational skflls. Hie compUcation arises in random assignment 
field research because staff must choose a treatment category for each individual before 
random assignment occurs and service begins. In many programs, espedaUy those that pride 
themsekes on individualized service plans that respond to the nxds of cUents, flexibility is 
important Staff value the option to adjust the service plan as client interests and needs 
change or as the original plan proves unsuccessful An up-front, binding designation of a 
service plan, even within the confines of fairly broad categories of services, can impinge on 
normal practices. Furthermore, in some JIPA programs with few service providers, the service 
recommendations are as much as function of the currently available services as the needs of 
the individual client If new options appear or old ones disappear, stafi" would normally change 
the service recommendation accordingly. 

3. These problems can be addressed by defining treatment categories that 
are anchored on a primary service but also allow participation in other, 
less intensive sendees as well. In addition, slan"can be offered the "safety 
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valve" of an nnrestricted service category for a snmU portion of those 
randomly assigned. 

Hie tension here is between a desire to estimate impacts for single activities, as was 
attempted in past nonexperimental studies of employment and training programs, or to examine 
more broadly defined categories of activicies that reflect the reahty of the JTPA assessment 
and service delivery system. In the JTPA study, with its goal of studying the program as it 
normally operates, the latter option was chosen. 

In the case of the JTPA study, classroom training in occupaUonal skills and OJT were 
obvious anchors for two treatment categories. Within each treatment category, cUents 
designated for these services and randomly assigned to the treatment group were also eUgible 
to receive other, less intensive services.^^ This greatly increased the ability of staff to foUow 
normal practices by giving them more flexibility in choosing services for individuals. 

This approach, however, carries with it a risk that is clearest for individuals recommended 
for the OJT activity category. In practice, when staff designate an activity category in 
experimental research, it means that they find the person appropriate for referral to that type 
of service, not that the person wiU necessarily receive it OJTs are the most difflcult type of 
activity to arrange because program staff must find an employer able to offer a type of traimng 
desired by the appUcant, negotiate an agreement to pay the employer to provide the training, 
and convince the employer to "hire" the specific appUcant; in many cases, each step is a 
challenge. Therefore, it is possible that a very high percent of those in the "OJT treatment 
group- wiU receive some JTPA services but that only 30 to 40 percent will actually be enrolled 
in an OJT.^^ 

Studies that use broadly defined activity categories must take care to document the nature 
of the services receKed by participants in each activity category so that impact findings can be 
interpreted intelligently. When this is completed in the JTPA study, researchers will be able 
to judge the appropriateness of the decision made in defining the activity categories. 



only real restriction was ihai those designated for classroom training in occupational skills could 
not be given an OJT, and vice versa. & r lu 

"in past experimental studies of OJT programs for welfare recipients, paniapation rates m OJTs 



were at this level 
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D. Lessons Aho yit Denying Services in CemimU 

In a voluntaiy, ongoing program like JTPA, denial of services to the control group is 
probably the major barrier to sites' participating in the study.^^ Several approaches have 
proved useful in addressing this issue. 

1. The ethical, political, and operational issues concerning denial of services 
should be raised and addressed by research staff expliciUy in early 
contacts with sites. 

Site staff appreciate frankness on this central issue. It is useful to emphasize the resource 
constraints the program faces and to clarify that the program is not an entitlement Offering 
technical assistance on client recruiting can bolster the argument that the study will not lead 
to a decline in the total number served. And presenting the long-term benefits of the study 
as a justification for the short-term costs also helps. 

2. The nature of the exclusion of controls can make « difference in how sites 
react, and the research perspective permits some flexibility in defining it 

The original JTPA study research design envisioned the type of control exclusion most 
difficult for sites to accept: a lengtiiy. s>•stem-^vide exclusion with no referral to alternative 
services. This is ideal from a research perspective because it allows tiie clearest test of tiie 
net impacts of a program. 

The research team and tiie Department of Labor adopted a number of modifications 
(listed below) that made the denial of services more palatable to sites. The first (shortening 
the period of exclusion) has the potential for lessening the difference in the level of services 
received by those in the treatment group and those in the control group. The second and 
third (excluding some persons from random assignment) mean that less than the fuU SDA 
program is mcliided in the study. TTie fourth (changing the random assignment ratio) reduces 
the statisu-cal precision of tiie impact estimates somewhat When the follow-up surveys and 
subsequent analyses are completed, the research implications of these changes will be better 
understood. The changes included: 



*Sues were less concerned about this in many past studies of mandatory programs because ihev were 
not denying their services to people who had actively sought them out There was less concern ioo m 
l^Kbp^^^ ^^^^ ^'^"y ^"tled demonstrations set up speafirally 'for 
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a. Shortening the period of esclosion from 30 months to 18 months while 
continuing follow-up of the entire research sample for 30 months and offering controls a list 
of other services in the community. These changes lessened sites' concerns that controls 
would feel cut off by the program with nothing in the way of help or that the public would 
find the study unethical Discussions with site staff indicated that most applicants for JTPA 
come to the program while unemployed and actively seeking work and that only a small 
percentage return in the future if they are not iniUally served. "ITiis meant that shortening the 
exclusion from 30 to 18 months could well have had little effect on the level of services after 
the 18-month exclusionary period ended. To lessen the chance of a high level of services after 
18 months, sites were prohibited from actively recruiting controls after that period ended; only 
if control group members returned to the program on their own could they be served. 

b. Excluding special types of applicants or types of activities from the study. The 
denial of services to certain types of applicants is especially troublesome for site staff because 
of their obvious need for assistance. When a class of applicants does not form a significant 
part of all participants, it is possible to exclude them from the study with little effect on the 
overall estimates of the program's impact In the National JTPA Study, some sites were 
allowed to exclude from random assignment homeless individuals, severely handicapped 
individuals, and court referrals to JTPA who needed to secure training to be granted probation. 
The same purpose can be served by excluding from the study specific parts of the program 
targeted on such applicants. 

c Providing the sites with a limited number of exclusions from random assignment, 
which they may use as they wish. Not al! troublesome cases can be anticipated in advance, 
so some flexibility is important In the JTPA study, sites were allowed to exempt from random 
assignment approximately 1 to 2 percent of their target sample. These exemptions had to be 
used before random assignment occurred, not to reverse an individual's assignment to the 
control group. Otherwise the exemptions would undermine the random assignment process. 

d. Reducing the percentage of applicants assigned to the control group. When the 
control group constitutes 50 percent of a given sample, statistical precision of impact estimates 
for that sample size is greatest The JTPA study initially planned such a 1:1 random 
assignment ratio. However, site concerns led to a change to two treatment group members for 



each control, which led to a modest reduction io statisUcal precision.^^ Changes beyond the 
2:1 ratio do reduce statistical precision much more, however. 

In the JTPA study, it was occasionally necessary to temporarily reduce the size of the 
control group to below one-third of the sample to address site concern* about reaching 
enroUment goals for specific groups; this occurred at some point during random assignment in 
five of the 16 study sites. In some cases, to meet enroUment targets in their contracts, service 
providers needed to serve more people than would have been available with a 2:1 raUo. In 
other cases, SDAs needed to enroll more of a particular group, such as youth, to meet state- 
imposed enrollment targets. Sites were typically willing to return to the normal 2:1 raUo after 
the specific problem was solved. This proved effective in keeping sites in the study, raising the 
total sample for analysis. 

E. Lessons on the Process of Random Assignment 

Avoiding logistical problems with random assignment is important because smooth 
procedures lessen the burden on site staff and eliminate operational "excuses" for not 
conducting random assignment Developing procedures to simplify the daily life of Une staff 
can build goodwill, a necessary ingredient in implementing a complex study. 

L Staff must be assisted to explain the study and to assure the informed 
consent of all members of the sample. 

Random assignment is unfamiliar to both staff and program applicants. It is important 
for the study that random assignment be explained clearly, consistently, and positively to all 
appUcants. Talking points for the explanation, scripts for key parts of the rules, and a 
videotape providing an explanation all proved useful, especially at the beginning of the study, 
when line staff were most uneasy about it and had not yet developed their ovra way of 
explaining it 

2. The process of conducting random assignment must be quick and easy for 
staff. 

Programs that process applications individually, as is the case throughout much of JTPA, 
may need to learn the results of random assignment immediately in order to begin 



^'ihj change frccii 1:1 lo 2:1 resulted in a 12 percent rcduaion in the statistical precision of the impaa 
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implementing tlie service plan whfle the applicant is stiU in the office. Even when the program 
processes applications on a "batched" basis, the random assignment process must be quick to 
avoid tying up stafE ToU-free phone numbers, computerized random assignment, and 
immediate feedback all help to provide a quick response. 

3. Line staff can be helped to adapt to changing procedures and the 
inevitable stress caused by participating in a complex study. 

Several sites in the JTPA study explicitly addressed the problem of staff stress by 
scheduling training sessions on managing change and stress. While the need for this wiU vary 
from site to site and may be different in voluntary and mandatory programs, it should be a 
permitted use of funds that are provided to support the implementation of the study. 

f"' Lessons on Measuring Participation in the Proy ram 

Measures of program participation rates are important in developing estimates of the 
needed sample size for impact estimates. Before a study begins, however, estimating 
participation rates for treatment groups is difficult 

1. Site staff often overestimate the proportion of applicants they serve. 
Therefore, it is important to monitor actual participation rates early in 
the study to see if the original estimates vv-ere accurate. 

The definition of participation rates used in impact estimates (the proportion of those 
who were randomly assigned to the treatment group and who actually participated in the 
program) has no counterpart in the normal operations of an agency. The closest estimate 
would be the proportion of eligible applicants who are enrolled, and this information is rarely 
collected by program staft Until a base of experience is built up, researchers must rely on the 
impressions of staff. In the JTPA experience, staff typically overestimated the percentage 
served because they remembered clients who enrolled and tended to forget those who dropped 
out of the intake process. 

in. A Final Lesson on Working with Sites 

Implementing a complex study is a collaborative effort v^"th the participating sites. The 
research staff must closely monitor the implementation of study procedures, since the actions 
taken to implement random assignment at the site level are the most critical part of the entire 
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research process. Continued close contact increases the probability that a problem will be 
discovered before it becomes severe. 

It is incumbent on the research teaai to initiate periodic contact with the sites, rather 
than waiting for them to raise issues. The research project wiU be a top priority of the 
research team, but surely not of the site staff; which contmue to have responsibility for the 
ongoing operaUon of the program. Furthermore, site staff may not be aware that a seemingly 
minor problem has major research implications. Conversely, site staff may struggle to follow 
a procedure that is presenting major operational difficulties without realizing that it might be 
modifiable. In some cases, researchers may be able to make changes that do not compromise 
the research objectives but make life much easier for partidpating sites. If the research team 
conveys a sense of reasonable flexibUity, site staff wiU explore options to address operational 
difficulties rather than concluding that their only recourse is to drop out of the study. 
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APPENDIX A 

OVERVIEW OF THE JOB TRAINING PARTNERSHIP ACT 



The Job Training Partnership Act (JTPA), Public Law 97-300, was enacted in 1982. It 
IS the principal federally funded program directed toward providing job training and 
employment services to economically disadvantaged individuals. 

Three of the five titles of JTPA autnorize direct service to participants. Tide HA, the 
subject of tills study, creates year-round programs for economicalty disadvantaged youth and 
adults and others with substantial barrien. to employment Titie IIB authorizes funding for 
summer youth programs. 



I. Funding Structure 

A. Fedfcral Funding bv Formula 

Under JTPA, funds for Titie HA are appropriated at the federal level and allotted by the 
U.S. Department of Labor (DOL) to the fifty states, tiie District of Columbia, and islands and 
temtones, accordmg to a formula. JTPA is almost entirely federally funded.^ 

State funding for Titie HA and IIB is then aUocated to sub-state areas (SDAs) according 
to the area's relative share of a three-part formula. This formula includes, in equal 
proportions: tiie relative number of economicaUy disadvantaged individuals,^ the relative number 



There are only two areas within JTPA in which funding matches are required. Section 202 (b)(1) sets 
aside 8 percent of Uie IIA allocatioii in which to cai.y out Section 123 of tiie Aa Section 123 specifies 
that Uie 80 percent of tiiese funds u;ed to provide services for eligible participants through cooperative 
educational agreements must be matched from funds outside tiie Aa The match may include the direa 
cost of employment or training services provided by state o local programs. 

Section 304 (a)(1) establishes a match requirement for Titie III. To qualify for financial assistance 
under this title, the state must demonstrate an expenditure of funds fi-om pubUc or private non-federal 
sour^ in an amount equal to tiie amount provided to the state under this title. 

An "economically disadvantaged individual' is defined as someone who: a) receives cash welfare 
payments under a federal, state, or local welfare program or is a member of a family that receives these 
payments; b) has a total family income (excluding unemployment compensation, child support payments, 
and welfare payments) for tiie six-month period prior to application which, in relation to family size, is not 
in excess of the higher of the poverty level established by tiie Office of Management and Budget (0MB) 
or 70 percent of the lower living standard; c) receives food stamps; d) is a foster child on behalf of whom 
sute or local government payments are made; or e) as permitted by regulations, is an adult mdividual with 
a handicap, whose own income meets tiie famUy income requirements, aclusive of tiie income of other 
family members. 



-137- 



180 



of unemployed individuals residing in areas of substantial unemployment,^ and the relative 
excess number of unemployed individuals.^ 



Funds for Title DA are further apportioned according to sub-parts, with speciflc target 
groups or types of services. States are mandated to allocate Title n funds as follows: 

Title JIA - 78 percent of the funds must be passed through to SDAs according to their 
relative share of the state total for each part of the three-part formula. 
Individuals eligible to participate in programs receiving assistance under this 
title must be economically disadvantaged, except that up to 10 percent of 
the participants in an SDA may be non-economically disadvantaged if tiiey 
have encountered barriers to employment 

8 percent is available for state Education and Coordination Grants to 
provide financial assistance to any state education agency responsible for 
employment and training. 

6 percent is available for the governor to provide incentive grants for 
programs exceeding performance standards, including incentives for serving 
hard-to-serve individuals. The funds are to be distributed among SDAs 
within the state in equitable proportions based on the degree to which the 
SDAs exceed their performance standards. If the full amount is uot needed 
to make incentive grants, the remaining amount is to be used for technical 
assistance to SDAs that do not qualify for the incentive grants. 

5 percent is avaUable for the governor for the cost of auditing activities, for 
administrative activities, and for other coordination and State Job Training 
Coordinating Council (SJTCC) activities, such as staffing for the SJTCC and 
meeting space. 

3 percent is avaUable for the training and placement of older individuals^ 
in employment opportunities with private business concerns. 

Title nB - 100 percent of the funds must be passed through to SDAs according to their 
relative share of the state total for each part of the three-part formula. 
These funds must be expended to provide education and employment 
services to economically disadvantaged youth over the summer months. 
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An 'area of substantial unemploymenf means an area of sufficient size and scope to sustain a Title 
IIA program and which bad an average rate of unemployment of at least 6^ percent for the most receni 
twelve-month period, as determined by E>OL 

^e 'excess number of unemployed individuals* means the number of unemployed individuals in excess 
of 4^ percent of the civilian labor force in the SDA or the number of unemployed individuals in excess of 
^fl^^^ ^ ^^'^ substantial unemployment in the SDA. 
Tlie term 'older individuals* means individuals age 55 or older. 
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A brief description of the purpose of the other titles of JITA follows: 



Title I - explains the state and local service delivery system and general program and 
administrative system under JTPA. 

TlUe in - establishes state-administered programs for dislocated workers and other 
target populations. 

Title IV - establishes funding for federaUy administered activities, including Native 
American programs, migrant and seasonal farmworker programs. Job Corps, 
veterans' employment programs, labor market information, and other national 
activities. 

TiUe V - amends the Wagner-Peyser and Social Security acts to foster coordination 
with the job training system. 

TJe multiple titles of JTPA with their overiapping population groups, sometimes resulted 
in an bDA serving the same mdividual in more than one title. Although Title HA is the focus 
of this study and is the largest single part of JTPA in terms of enrollment and budget, other 
stX of TiUe^aT ""^"^'^ "^^^ "^'^^ HA in ways that complicate the implementation of a 

T- ^^^^^ frequently move between Title HA and Title UB, and between TiUe IIA and 
TOe in. This matters because controls are restricted from receiving any SDA-administered 
JTPA services in the area. (State-administered programs, which caused some titles or sub- 
parts to be operated by agencies different from those administering Title HA in some states 
increased the problems of coordinating the implementation of the study.) 

Title nB, the summer youth program, at limes recruits youth out of IIA, and vice versa- 
HA at tunes reenrolls youth at the end of the summer. (All Title HB youth are eligible for 
Title HA, although the reveree is not always true because of age and income limitations in 
Title HE.) Title m, serving dislocated workers, may move people into UA if Title m funds 
have been fully encumbered in a program year. 

B. Expenditure Requirements for TH!e nA 

Cost categories: JTPA cost categories were established to ensure that a 
significant proportion of the funds would be directly expended for employment and training 
activities. The cost categories imposed a 15 percent limit on administrative racpenditures (down 
from 20 percent under CETA). The cost categories also require that not more than 30 percent 
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can be spent on both administraUon and supportive services combined,^ lcj<ving at least 70 
percent of the allocation for training. 



2. Restrktioas on income support payments: In defining allowable costs, the Act 
ehminated the payment of stipends, or aUowances fnr participants in classroom training 
activities. The Act and r?gulations also defined the cost categories for work experience in 
such a way that it effectively restricted its use. The Act eliminated all subsidized public service 
emplqyment 



n. Eligible Pop ulflrtnn 

The purpose of JTPA is to prepare youth^ and unskiUed adults^ for entry into the labor 
force. JTPA authorizes the provision of job training for economically disadvantaged individuals 
and others facing serious barriers to employment, including those who aie in special need of 
such training m order to obtain productive employment JTPA is not, however, an entitlement 
program: Access is not guaranteed to all who satisfy the eligibility rules. Section 141 of th-. 
Act specifies that the service delivery system is to prx)vide employment and training 
opportumties to those who can benefit from, and who are most in need of; such opportunities 
The system is also expected to make efforts to provide "equitable services" among "substanUal 
segments" of the eUgible population. These two terms are difGcult to define, and states 
interpret them differently. 

State and SDA interpretation of these two requirements could result in a more or less 
narrowly defined target population. 

1. Youth: The Act requires that not less than 40 percent of TiUe HA 78 percent funds 
must be expended to provide services to eUgible youth- To achieve a 40 percent level of 
expenditure, SDAs might have to enroU more or less than 40 peroisnt of the youth population 
depending on the average cost per youtfci served. 

2. AFDC recipients and school dropouts: Recipients of Aid to Families with 
Dependent Children (AFDQ who are required to participate in work programs and eligible 
high school dropouts are to be served on an equitable basis, taking into account their 
proportion of economically disadvantaged persons 16 years of age or over in the area. 
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PIC may request a waiver of this limitation by demonstrating that excess costs result from one 
of the following circumstances: a) the unemployment rate acceds the national average by at least 3 
percentage points and the ratio of currem private employment to population in the area is less than the 
national average of such ratio; b) the area plans to serve a disproportionately high number of partiapants 
rcqunng exceptional supportive services, such as individuals with handicaps; c) the cost of providing 
necessaiy child care exceeds one-half of all supportive services in the SDA; d) the costs of providing 
necessary transportation exceeds one-ihird of all support^e services in the SDA, or e) a substantial portion 
of participants are in training programs of nine months' duration or more. 
T^e term ■youth" means an indiviY-ua: who is age 16 through 21. 
^e term "adult" means an individual who is 22 years of age or older. 
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3. OJher tai^get groaps: SDAs are also to make eCforts to provide equitable services 
to other substantial segments of the eligible population. •Substanti'al segments" are defined at 

«r!u.*^^t.°' ^^^^ ^"'^ ^ levels of intensity across states 

Within these parameter, SDAs decide whetiier to include programs for in-school youtii in 
their job training plans. 

rrs.inf^nJL<t^ ^^""^'y Pi°g^^ Unlike tiie new Job OpportuniU'es and Basic Skills 
irammg (JOBS) Program and some of its predecessor welfare-to-work programs, JTPA does 
not require people to participate as a condition of receiving a direct personal benefit, such as 
a cash grant As noted above, JTPA is not an entitlement program, such as the public 
employment service. F"""^ 



ni. Adminirtrat{ ve/Overs{f ;hf Sfnirf,..Y> 



n.r.n,^« f^. ^"'^ ^'^^ "^^^ ^"^^^^ select, witilin legislated 

^'^.r^trative/oversight structure for the delivery of services and the Lcs of 

T^^A ^V^^^''"^ ^'"^^ I^e' the persons and organStions 

S^e^trc or.^H-'" ''""'"^ structure Sosen meant that many 

different actors with diverse perspectives were involved in making decisions, such as Uie one 
concemmK whether to participate m the National JTPA Study. 

A. Designation of SDAs 

f^y^A^^'^''f°'' °^ ^f"^' ^ '^^^ by Service Deliveiy Areas 

(SDAs). The designation of SDAs is a responsibiUty of the governor of each state based on 
a proposal from the SJTCC appomted by the governor. SDAs provide the vehicle for 
deUvenng job traimng services to the local areas. 

SDAs are generally geographicaUy defined as a large unit of local government They 
must comprise all of a state or one or more units of local government, and they must be 
consistent with labor market areas or standard metropoUtan statistical areas or with areas in 
which related services are provided under other state or federal programs. 

^' Variety of L ocai Admtnistrative/Oversight Structures 

JTPA was a landmark piece of legislation because it increased the role of the states and 
established an equal partnership between the public and private sectors in the planning and 



for of exemplaiy youth programs, including one that allows for preemployment skills training 

for individuals aged 14 and 15. are described in Seaion 205 of the Act ^ 

a.n.r.nJl?"' ^PP'°^^ designation as an SDA from units of 

S ot^i^Tfil?''"""'"' wiih populations of 200.000 or more or fron. any cx)nsortiun, of cx)nuguoSs unm 
of general loca govemmeni wiih aggregate populations of 200.000 or more thai serve a substantial part of 

PrommTSpT JJifJIfr'Jf"'' '"'^ '^^""'^ f™"" Concentrated Employmen, 

A^T^aT ^ ^ ^^'^^ ""^^'^ Comprehensive Employment and Training 
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oyereight of a federally funded job training program. The Act passed substantial authority for 
the f^rally funded JTPA system to the governor. DOL, in keeping with the general trend 
toward decentrdization that led to JTPA's structure, has reinforced the statutory provisions 
■ that provide authonty to the states by minimizing system-wide directives (including adding onlv 
a mmimal number of regulations to the Act). 

The governor designates the state agency responsible for the oversight of aU JTPA 
programs m the state, appoints members to the SJTCC, conducts state programs, desimates 
SDM and approves SDA plans. The SJTCCs recommend poUcy and advise the governor on 
the designauon of SDAs, on approv'^ of SDA plans, and on program implementau'on. 

^ch SDA is required to establish a Private Industry Cbuncfl (PIC). The majority of 
^r/ , J"^^ ^ coEposed of representatives of the private sector. Six other groups, 
mcludmg kbor. educaUon. community-based organizations (CBOs). representatives of indMdu^ 
win nandicaps the pubhc employment service, and economic development agendes. must also 

SL[dSt!"tf vfn'^ll^^.- '^^ °^ representaUon by additional groups is 

decided by the PIC The diverse representaUon on the PIC means that varying interests are 
heard and considered in the decision-making process. 

T.. "® appointed, with the approval of the governor, by the Chief 

Elected Offiaal(s) - CEOs - of the um^s) of local government from a list of nominees from 
busmess, educaUon, labor, and other designated organizations defined in tiie Act The CEOs 
submit Uieir recommendations for PIC appointments to the governor for approval 

Once tiie PIC is established, the PIC and CEO joinUy select the planning entity, grant 
recipient, and administrative entity. Hie planning entity is responsible for developing tiie 
biennial Job Training Plan, which defines how the fimds will be spent (Modifications are 
requir«l annually.) -nie grant recipient is tiie organization that receives tiie federal JTPA 
funds from tiie state. Tne administrative entity is tiie organization responsible for carrying 
out tiie plan. One or more organizations can be designated to fulfill each of these functions. 

C Variety in Provisfon of Services 

Within a wide range of authorized employment and U-aining services, SDAs select the mix 
of programs to be provided and organize the delivery structure for each of tiie components 
SDAs become brokers of services, working with and tiirough otiier human resource delivery 
systems to a greater or lesser extent TTiey typically do not provide most services directly. 
Instead, SDAs conU-act witii otiier agencies to provide services. Some SDAs contract out for 
the majonty of services, including the determination of program eligibility, and others provide 
aU but vocational skflls training components with the. own stafil In aU cases, the flow of 
individuals from one agency to another necessitates substantial levels of coordination, but the 
importance of carefully coordinated services is compounded in systems with multiple intake 
points and service providers. 
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I^* Tlie Federally Mandate d Performance Stendflni Sy stem 

The Act requires the U.S. Secretaiy of Labor to establish standards based on factore 
that may include placement in unsubsidized employment, retendon in unsubsidized employment, 
mcrease in earnings, reduction in the number of cash welfare recipients, and reduction in the 
amounts of case welfare payments. For youth, employment competencies, school compledon, 
and enrollment m other training programs or enlistment in the miUtary are also to be 
established. Standards relating program expenditures to various performance measures also are 
required. 

Governors may prescribe variations in these standards. Whfle a few states have adopted 
the benchmarks established by DOL, nearly aU have adjusted the national benchmarks based 
on local factors, using a regression model developed by DOL. Some states have added 
standards, and a few have developed alternative performance standard systems. Any such 
alternative system must be approved by DOL. 

The importance of doing weU on the performance standards is reinforced by the 
distribuuon of TiUe HA 6 percent incentive funds (discussed in Section L\ of this Appendix). 
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APPENDIX B 
PROFILES OF THE SITES 



This appendbr opens with several tables summarizing information about the 16 service 
delivery areas that are participating in the National JTPA Study. It follows with a set of 
profiles of the sites describing, in each case, the site's locadon, size, population, and labor 
market; its program (structure, services, parricipant characterisdcs, and performance on Title 
HA programs); and its implementation of the study. 

These profiles are intentionally brief and are meant to provide the reader with some 
understanding of the diversity across sites. Programs not included in the sample, such as Titles 
m and UB, and services to in-school youth thro^tgh Title HA, are not specifically discussed- 
More comprehensive information on the sites will be provided in future JTPA reports. 

To ensure a coa':ttent base for comparison across sites, 1979 census data were the 
-primary source of informadon on populadon and size, whfle the JTPA Annual Status Report 
(JASR) for 1986 were used for information on chent characteristics and performance.^ SDAs* 
Annual Job Training Plans, and observations and data collected by the resej jchers during the 
implementarion of the study, were also drawn on in developing these profiles. 



The JASR data contain information on individuals 'terminated* from the program each program year. 
Terminated' is the label assigned to people who were enrolled in JTPA at some time dunng the year and 
whose enrollment ended because of employment or program completion, or because they dropped oui or 
were removed from the program early. 
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TABLE B.l 

LOCAL ECONOHIC CONDITIONS, BY PARTICIPATING SDAs 



SDA 


Un fifliD 1 0 vmen t 
Rate 
(Progranj 
Year 1986) 


f^vci owe 

Wage 
(Calendar 
Year 19861* 


rcrcent ot 
Families 
Below 


Population 
Density 
(# of Persons 
per oquare ni it) 


Capital Area, KS (Jackson) 


6.7 


17 200 




310 


Concentrated Employment 
Program, HT (Butte) 


7.7 


Ifi 700 


7 ? 


10 


Coosa Valley, GA (Rome) 


5.7 


15,300 


11.2 


80 


Corpus Christ! /Nueces County, 
TX 


10.4 


18.500 


13.4 


3Z0 


Crawford/Hancock/Harion/ 
Wyandot Counties, OH 




Tft inn 


/ .2 


120 


East Central Iowa (Cedar 
Rapids) 


5.9 


17,200 


6.1 


90 


Greater Omaha, NE 


5.6 


17,700 


6.6 


520 


Heartland, FL (Lakeland) 


10.3 


15 200 

,b W 




OA 


Jersey city, HJ 


6.6 


20,400 


14.7 


7,000 


Larir*>er County, CO (Fort 
Collins) 


5.8 


17,500 






Hacon/De Witt Counties, 
IL (Decatur) 


11.3 


22,300 


7 a 


19U 


Northeast, IN (Fort Wayne) 


6.0 


17,800 


5 9 


IQvl 


Northwest, MN (Crookston 
and Thief River Falls) 


10.,0 


13,600 


11.1 


10 


Oakland, CA 


8.7 


22,000 


8.7 


6,300 


Ptt)vidence/Cranston, RI 


4.6 


17,000 


9.0 


4,630 


Springfield, HO 


5.9 


15,200 


10.1 


70 


Average 


7.8 


17,605 


9.4 


1,245 


National Average 


8.0 


16.900 


9.4 


600 



(continued) 
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TABLE B.l (continued) 



SOURCES: JTPA Annual Status Report (JASR) for program year 1986. The JASR data on percent 
of famiHes below the poverty level are based on 1979 census data; population density is 
based on 1980 census data. 

NOTES: "Average wage is the total payroll reported to federal and state unemployment 
insarence programs in the SDA divided by the number of employees in the SDA. 

The poverty level was defined in 1979 as $7,356 for a family of four with two 

children. 
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TABLE B.2 



DEMOGRAPHIC CHARACTERISTICS OF JTPA TERHINEES FROM TITLE IIA (1986) 

BY PARTICIPATING SITES 



Site 



Total 



Capital Area, HS 

(Jackson) 77] 

Concentrated Employment 
Program, HT (Butte) 731 

Coosa Valley, GA (Rome) 464 

Corpus Christi/Nueces 
^ County, TX 993 
»— • 

00 Crawford/Hancock/Harfon/ 

Wyandot Counties, OH 673 

East Central Iowa 

(Cedar Rapids) 73I 

Greater Omaha, HE 853 

Heartland, FL 

(Lakeland) 3^044 

Jersey City, NJ i,213 



Larimer County, CO 
(Fort Collins) 



170 



Adult 



41 

64 

52 

51 

47 

50 
55 

63 
51 



Youth 



59 

36 
48 

49 

53 

50 
45 

37 
49 



Hale 



50 

34 
36 

47 

55 

54 
41 

44 
43 



Female white 
(%) (%) 



50 

66 
64 

53 

45 

46 
59 

56 
57 



15 

92 
90 

19 

94 

89 
40 

61 
8 



o, . American Limited 

Black Hispanic Asian Indian English 
^*) W (V) {%) 



84 



1 
9 



6 
54 

35 
69 



2 
1 

72 



0 
0 



4 

0 

0 
2 



5 
0 



1 
1 

0 
0 



0 
0 



3 
1 

1 
5 




1 2 
(continued) 
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TABLE B.2 (continued) 



Site Total 

Hacon/De Witt 

Counties, IL (Decatur) 621 
Northeast, IN 

(Fort Wayne) 1,726 

Horthwest, m (Crookston 

and Thief River Falls) 408 

Oakland, CA 934 

Providence/Cranston, RI 639 

1,117 



I 

S Springfield, HO 



Average for Sites 957 
National Average 1,318 
SOURCE: JTPA Annual Status Report (JASR) for program year 1986. 



Adult 
(%) 



58 

66 

57 
61 
54 

/fiO 
56 
56 



Youth 
(*) 



42 

34 

43 
39 
46 
40 
44 
44 



Hale 
(%) 



59 

47 

56 
54 
52 
50 
48 
47 



Female 
(%) 



41 

53 

44 
46 
48 
50 
52 
53 



White 
(%) 



64 

68 

97 
6 
38 
94 
60 
61 



, ,„ American Limited 

Black Hispanic Asian Indian English 
(*) (») (%) (%) (%) 



36 
27 

0 
68 
34 

2 
27 
25 



1 
4 
19 

0 
9 
9 



0 
21 
8 
3 
3 
3 



I 
0 
I 
1 
1 
2 



1 

0 
14 
10 

3 

5 
6 
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TABLE B.^ 

ACTUAL PERFORMANCE FOR JTPA TERHINEES FROM TITLE IIA 





Adults 


Youth 


Site 


Entered 
Employment 
Rate (%)" 


Cost per 
Entered 
Employ- 
ment {$)'' 


Average 
Wage at 
Place- 
ment {$)C 


Welfare 

Entered 
Employment 
Rate (%)<* 


Positive 
Termi nation 
Rate (%)* 


Youth 
Entered 
Employment 
Rate (%)^ 


Cost per 
Positive 

Tenni nation ($)9 


Capital Area, HS (Jackson) 


78.4 


2.184 


4.41 


76.5 


94.7 


48.5 


1.354 


Concentrated Employment 
Program, HT (Butte) 


74.6 


3,249 


5.55 


69.0 


87.2 


68.8 


2.217 


Coosti Valley, GA (Rome) 


89.6 


2,182 


4.75 


89.3 


81.7 


43.8 


1.945 


Corpus Christi /Nueces County, 
TX 


77.0 


2,913 


4.77 


65.4 


80.2 


46.0 


2.645 


Crawford/Hancock/Harion/ 
Wyandot Counties, OH 


57.0 


4,813 


4.76 


55.7 


76.6 


38.2 


3.487 


East Central Iowa (Cedar 
Rapids) 


82.1 


2.618 


5.46 


72.5 


86.4 


62.0 


2.292 


Greater Omaha, NE 


68.0 


3,397 


4.86 


54.0 


84.0 


46.0 


2.995 


Heartland, FL (Lakeland) 


75.0 


2,523 


4.69 


70.7 


82.8 


58.3 


2.468 


Jersey City, NJ 


92.7 


2,504 


5.19 


92.6 


95.1 


68.2 


2.652 


Larimer County, CO 
(Fort Collins) 


74.4 


1,773 


4.65 


55.4 


76.6 


60.9 


2.258 
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TABLE B.3 (continued) 



Site 



I 



National Average 



Cost per 
Entered Entered 
Emp 1 oyinen t Emp 1 oy- 
R«te ment 



Average 
Wage at 
Place- 
ment 



Welfare 
Entered 
Employment 
Rate (%)^ 



Hacon/De WUt Counties, 


















68.7 


3,213 


5,15 


69.9 


77.2 


30.0 


2,347 


Northeast, IN (Fort Wayne) 


91.0 


1,504 


4.82 


87.0 


87.0 


72.0 


2.962 


Northwest, MN (Crookston 
















and Thief River Falls) 


85.0 


2,553 


5.24 


66.7 


87.9 


42.0 


1.566 


Oakland, CA 


68.9 


3,258 


6.35 


60.8 


83.4 


41.0 


3,110 


Providence/Cranston, RI 


75.1 


4,424 


5.37 


72.1 


87.0 


57.3 


2,247 


Springfield, HO 


86.0 


2,292 


4.63 


79.0 


94.0 


55.0 


1,724 



72.4 



2,952 



5.08 



63.7 



Youth 

Positive Entered 
Termination Employment 
Rate (%)e Rate (%)f 



80.8 



52.6 



Cost per 
Positive 

Termination ($)? 



2,392 



2,427 



SOURCE: JTPA Annual status Report (JASR) for program year 1986. 

NOTES: «The percent of all adult tsminees who entered employment. 

Total expenditures for adults divided by the number of adults who entered employment 
^Average hourly wage of adults who entered employment at temination. 

'JHe ZllTt ll 1\\ "^""It °" ^"^'^ ^^'^ P'-^S'--'") entered employment. 

employabin;;^nl^"e:e:: ;:n:ul::™""^ - ^^^^^^ ^-^-^^ -plo^ent^r^et one of the 

The percent of all youth teminees who entered employment. 
«Total expenditure, for youth divided by the number of youth who had a positive temination. (See footnote e ) 
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TABLE B.4 

DIFFERENCE BETWEEN ACTUAL PERFORMANCE LEVELS AND STATE 
PERFORMANCE STANDARDS FOR JTPA TITLE IIA TERMINEES (1985) 
BY PARTICIPATING SITES 




S1t« 

Capital Area, MS (Jackson) 

Concentrated Employment 
Program, HT (Butte) 

, Coosa Valley, GA (Rome) 

Corpus Chrlstl /Nueces County, 
TX 

Crawford/Hancock/Harfon/ 
Wyandot Counties, OH 

East Central Iowa (Cedar 
Rapids) 

Greater Omaha, NE 

Heartland, FL (Lakeland) 

Jersey city, NJ 

Larimer County, CO 
(Fort Collins) 



Cost per 
Entered Entered 
Employment Employ- 
Rate (%)a ment (%)•> 



15.4 



-1.4 

14.5 
6.0 
11.8 
36.0 

15.0 



-812 



12.6 -1,125 
21.4 -1,373 

14.9 -2,839 



-963 

-1,699 
-977 
-1,167 
-2,990 

-2,785 



Average 
Wage at 
Place- 
ment ($)C 



Welfare 
Entered 
Employment 
Rate (%)«* 



0.23 


26.6 


21.5 


15.3 


0.64 


18.0 


12.2 


25.8 


0.29 


31.3 


11.5 


0.0 


0.07 


21.1 


12.8 


3.7 


-0.17 


7.6 


-5.7 


2.6 


0.68 


16.3 


7.2 


13.5 


0./2 


3.0 


9.0 


3.0 


0.73 


14.5 


8.1 


23.8 


0.48 


42.7 


28.5 


28.8 


-0.28 


6.5 


7.8 


22.8 



Youth 

Positive Entered 
Termination Employment 
Rate (%)* Rate (%)^ 



o i7> 
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Cost per 
Positive 

Termination ($)9 
-1,641 



-2,683 
-841 

-2,309 

48 

-1,644 
-1,905 
-415 
-1,557 

-2,234 

(contl^ued)j 



TABLE B,4 (continued) 



I 





Adults 


Youth 


Site 


Entered 
Empl oyment 
Rate {%)« 


Cost per 
Entered 
Emplov* 
ment (%)^ 


Average 

Pi ATP* 

• 1 aL.c— 

ment ($)C 


Welfare 
entered 
rrif/ioynient 
Rate (%)^ 


Positive 
Tennination 
Rate (%)® 


Youth 
Entered 
Employment 
Rate (%)^ 


Cost per 
Positive 

Tennination ($)9 


Hacon/De Witt Counties, 
















IL (Decatur) 




-2,516 


-0.19 


16.7 


1.6 


0.5 


- -611 


Northeast, IN 
















(Fort Wayne) 


22.0 


—1 , / oo 


0.20 


33.0 


7.0 


14.0 


-707 


Northwest, HH (Crookston 
















and Thi^f Pivpr PaIIc) 


15.2 


•1,320 


0.53 


9.0 


5.1 


-2.2 


-1,829 


Oakland. CA 


6.8 


-1,631 


1.32 


6.8 


13.9 


8.6 


-981 


Providence/Cranston, RI 


16.5 


•8 


0.46 


25.4 


12.0 


14.0 


-2,653 


jpr 1 ii^ 1 1 c 1 U , nu 


14.0 


-1,526 


-0.02 


22.0 


14.0 


7.0 


-1,626 


Average for Sites 


14.3 


-1,595 


0.36 


18.8 


10.4 


11.3 


-1,474 


National Average 


10.8 


-1,510 


0.44 


12.2 


6.9 


12.4 


-1,334 



JTPA Annual status Report (JASR) for program year 1986. 
NOTES: "The percent of all adult termfnees who entered emplov-.nr.nt. 



»1 

Total expenditures for adults divided by the number of adults who entered employment. 
^Average hourly wage of adults who entered employment at termination. 

eT 



The percent of all adult tenninees on welfare (when they entered the program) who entered employment. 

^pioyabiiitrenj:nc:::nrd:i!nfu;^s!^"'"" " '''''' ^"^^^^^ - - °^ 

The percent of all youth tenninees who entered employment. 
fTotal expenditures for youth divided by the number of youth who had a positive termination. (Sue footnote e.) 
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CAPITAL AREA TRAINING \m EMPLOlTVfENT r o NSORTTTIM rT»rW Misstsrfp pn 
Study rontart 
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Three oties in northcep.tral Mississippi are served by the Capital Area Training and 
Employnient Consortium. They are Jackson. Clinton, and Pearl, with Jackson having 85 
percent of the total populaUon of 239,000. lhirty.eight percent of the SDA's populaUon is 
black and other nunonties. An estimated 13 percent of die famiUes had incomS^below the 
fedenj] poverty level in 1979. 

The unemployment rate was 6.9 percent in 1985 and 6.7 percent in 1986. The largest 
source of jobs is the service sector. The state capital in Jackson provides whiteK»Uar 
emp oymenL Other serice industries, such as hospitals, are another major^l^^ of 
employment The average wage in the SDA was $17,200 in program year 1986. 

The Program 

Hie SDA is a consortium, with the city of Jackson serving as the administrative and 
pknnmg enUly and grant recipient for the program. Staff; employed by the city in the 
Department of Human and Cultural Services, are housed in one central office. Hie SDA is 
the grant recipient for IlUe HA and JIB funds, but does not operate Title HA 3 and 8 percent 
programs or Title m. r <=. ptit^ui 

The SDA offers classroom training in a variety •: " occupational areas. Performance- 
based contracts are used to fiind training in specific vocational areas such as health occupations 
auto mechamcs, truck driving, cooking, and security. JTPA is the primary source of businesJ 
for a number of these contractors. Other contractors, including a community coUeee and 
Jacteon State University, are funded to provide other activities, including a 6- to 8-week iob 
c ub/employability development program, which can lead to an OJT, and ABE and GED 
^ , * ^^^^ contractors were utUized during the study period In addition, SDA staff 
develop OJT paitions and place participants in employment Some of the OJTs are developed 
for empioyer-referred candidates. ^ 

Of the 771 people terminated &om Title HA in program year 1986, about 60 percent 
were youth, v^hose tune in the program averaged 14 weeks. Nineteen percent of the youth 
were high school dropouts and 30 percent were AHX: recipients. About half the youth were 
age 18 or over. Adults, representing 41 percent cf the terminees, averaged 5 weeks in the 
program. Nmeteen percent were high school dropou-^ and 11 percent were AFDC rec-pients 
Of die total ierminees, 84 percent were black and 15 percent were white. Perf-ormance 
standards were exceeded by the SDa during this year. 

Study Implementation 

Service pro-^/iders are responsible for their own recruitment. The SDA har experienced 
^!^t.!'^^ '"''^'^S enrollment targets for the white population. Sixty percent of the 
eligibility interviews are conducted by city JTPA staff at the SDA office, and 40 percent are 
held at contractor sites prior to the start of new classes. Some applicants are screened by the 
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provider and then sent to the SDA office for the eligibHity interview. When eligibility can be 
determined without additional follow-up during the initial interview, the random assignment 
phone call takes place immediately. Otherwise, random assignment occurs after additional 
documentation has been obtained, without the applicant being present After random 
assignment, those assigned to the treatment group are referred to the service providers, who 
determine whom they will enroll 

A variety of release and information forms, in addition to those required for the study, 
are required the state for those enrolled in the program. Several months after /mdom 
assignment began, procedures were modified so that these additional forms were corjpleted 
only for individuals assigned to the experimental group. 

Ihe start-up of the study in February 1988 coincided with the release of additional funds 
to the SDA by the state. Therefore, the SDA had to issue additional Requests for Proposals 
and quickly im'tiate new programs just as assignments to the control group were beginning to 
occur. The timing of these two events created some problems for the SDA, and a performance 
adjustment was eventually needed over two program years, with random assignment ending in 
March 1989. Nevertheless, the SDA met its random assignment sample goal several months 
early and extended random assignment for one month in order to increase the sample above 
the original goal 

During the period of the study, 41 percent of the apph'cants were recommended for 
classroom occupational skills training, 37 percent for other services, and 22 percent for OJT. 
Individuals served through a contract with vocational rehabilitation services were excluded 
from random assignment 



CONCENTRATED EMPLOYME?^ PROGRAM ARE\ (Butte and Helen a. Montenfl) 
Study Context 

The Concentrated Employment Program (CEP) serves ten counties in western Montana. 
The CEP SDA has approximately 125,000 residents, of whom about 37,000 live in Butte; 24,000 
live h Helena. The remaining parts of the SDA are very rural, with no towns over 15,0(K) in 
population. SDA residents are primarily white, and 7.5 percent of all families had incomes 
below the federal poverty level in 1979. 

The SDA's economy has traditionally been based on mining, timber, and ranching. In 
the late 1970s and early 1980s, major layoffs in mining and related industries occurred in Butte 
and Anaconda, causing a decline in the local economy in these areas. The state capital in 
Helena provides white^x^Iiar employment, while Butte's economy is gradually diversifying. 
Unemployment in the SDA was 9 percent in 1984 and gradually declined to 7.4 peicent in 
1987. The average annual wage for workers in the SDA was $16,700 in program year 1986. 
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Hie Montana Department of Labor and Industry, Employment Policy Division, is the 
grant reapient for the SDA, which has a Private Industry Council appointed by the CEP 
Council of County Commissioners. The Council of County Commissioners also is consulted 
on JTPA poUcy and at times meets jointly with the PIC The JTPA program is operated by 
the Montana Job Service Division, under contract with the Employment Policy Division. Four 
Job Service offices provide JTPA services: In iielena and Butte, separate JTPA intake offices 
serve clients, whfle in Anaconda and Dillon, the Job Service provides both its usual job listing 
services and JTPA in a single office. * 

The Job Service staff offer job search assistance and OJI in all offices and provide 
classroom training by referrals to other agendes. In Helena, a basic skills brush-up course 
foUowed by clerical training was an important activity, preparing people for employment in 
the capital area, while in much of the rest of the SDA OJT was the most common activity. 
In the rural parts of the SDA, the nearest classroom training service provider was many mUes 

Of the 731 people terminated from Title HA in program year 1986, 64 percent were 
aduUs. Among adults, 9 percent were high school dropouts and 28 percent were receivin-- 
AFDC; for youth, the figures were 5 percent and 16 percent, respectively. Average time iJ 
the program was 21 weeks for adults and 18 weeks for youth. 

The SDA met all its performance standards in program year 1986. 
Study Implementation 

The normal intake procedures in the two larger offices (Helena and Butte) differed 
somewhat from those in the smaller offices. In Helena and Butte, intake for JTPA was 
normally done on a group basis, typically scheduled once a weeL Oients completed the JTPA 
apnUcation and met with staff for a brief assessment interview. An orientation session was 
then held for appUcants who were eligible and appropriate for JTPA At this session, staff 
would provide job counseUng, conduct a fuUer assessment, and then work with individuals to 
arrange ser.ices. In the smaller offices, staff worked with clients individually and the 
procedures could be handled more informalfy. 

In order to introduce the study procedures in this SDA three issues had to be addressed 
FiRi, m the two larger offices, an appropriate point for the designation of the activity category 
had to be identified that would come late enough to provide staff with information on the 
clients but before services were provided during orientation. Staff agreed to base 
recommenddtions for services on a review of the materials assembled during the group intake 
sessions. At these sessions. appUcants would complete the background information form and 
mformed consent foni! and random assignment was then conducted. Only those randomly 
assigned to the treatment group would be scheduled to attend orientation and then be referred 
to service providers. 
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Second, some cb'ents who were interested in an OJT would tty to arrange one on' their 
own; this was espedaUy common in the smaller offices. Typically, the percentage of individuals 
who sought and found an OJT in this way was quite low, so randon assignment could not 
come before the beginning of this individual OJT search. However, individuals needed to 
understand that if they returned with a possible OJT, enroUment in JTPA was not guaranteed. 
As was ahvays the case, .their eligibility would have to be redetermined, and staff wo" ve 
to judge the appropriateness of the job for an OJT, and - during the period of the study - 
they v.'ould have to go through random assignment. 

Third, in Helena and Butte, agencies serving displaced homemakers provided pre- 
employment skills training and counseling, using state funds, and often then enrolled some of 
their participants into JEPA-funded clerical training. Since the iniUal portion of the sequence 
was not funded by JTPA and only a portion of participants made it to the JTPA-funded 
activities, cUents referred from these service providers were excluded from the study. 

The SDA began random assignment in July 1988 and completed it in September 1989, 
fallmg somewhat short of its original target sample. 



COOSA RF GIONAL nEVF7^PMENT CENTER (Rome. Genr^ a) 

Study Context 

The Coosa Vallqr Regional Development Center provides JTPA services for a ten-county 
area in the northwestern part of Georgia. The SDA is large and rural, and pubUc 
transportation is nonexistent except in Rome, which is the largest city, with a population of 
30,000. Services are provided through a decentralized structure in order to reach the total 
SDA population of 355,000, which is predominantly (more than 90 percent) whits. 
Approximately 11 percent of the families had incomes below the federal poverty level in 1979. 

The most important employment sector is manufacturing, which employed 38 percent of 
the workforce in 1980. The textile and apparel industries dominant this sector. The service 
set .or also supplies a significant number of jobs, employing 19 percent of the workforce. The 
unemployment rate started to decline just as random assignment began - from 7 percent in 
1985 to 5.7 percent in 1986. The average wage in the area was $15,300 per year in proeram 
year 1986. ' ^ ^ 

Th" Program 

A state-created but locally administeret' r:egional Development Center (RDC) is the 
planning and adminisi.ative entity and grant recipient for the SDA RDQ fulfill multiple 
labor-market-related functions in Georgia and receive funds from the state as well as from 
the programs they oversee, such as JTPA The Coosa Valley RDC provides fiscal and program 
oversight and contracts out all services. The PIC, whose members include a number of the 
JTPA contractors, provides input to the RDC. The RDC coruracted v^th Beny College to 
provide intake services as well as occupational training. Because of the large area served by 
the SDA, intake was conducted at provider sites and in central public locations wiihin the 
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counda; (e.g., courthouses), as weU as at Berry CoUege's JTPA office. FoUowing the 
determination of eligibility for services, Berry College staff made referrals to the 18 JTPA 
contractors that dealt with the population covered by the study. 

Community colleges, vocational coUeges, technical schools, and private for-proGt and not- 
for-profit organizations provided classroom training in a wide range of occupations. 
Community-based organizations worked primarily with out-of-school youth in GED preparation, 
job search activities, and the development of youth competencies. The pri\'ate contractors also 
arranged OJTs. Contracts for all of the above activities were primarily performance-based. 

Of the 464 persons terminated from TiUe HA of JTPA in program year 1986, 52 percent 
were adults and 48 percent were youth. Youth averaged 19 weeks in the program and adults 
averaged 14 weeks. Forty-one percent of the adults were high school dropouts and 12 percent 
were ATOC recipients at the time of appUcaUon to JTPA. Seventeen percent of the youth 
were high school Oxopouts and 16 percent viere AFDC redpients. 

In program year 1986, the SDA met aU its performance standards. 
Study Implementation 

Random assignment started in January 1988 and ended in August 1988. Recruitment 
was not a general problem, although some OJT providers said that the study caused them to 
lose good candidates. Over the study period, there were several organizational changes that 
compUcated the unpleraentation process. Berry CoUege had only recently become the 
contractor for intake services and was still refining procedures at the time the study started. 
The good relationship that developed betweer. the service providers, Berry CoUege, and the 
SDA staff was critical to the successful implementation of the study. The need to recommend 
individuals to an activity category prior to random assignment required further adaptations ior 
service providers not providing classroom training. Berry College certified and referred 
applicants to the providers, who were asked to determine whether they would accept the 
applicant Most classroom training contractors were already doing this type of assessment. 
Once the client was found appropriate for services by the contractor, Berry College would 
complete the random assignment call and then inform both the contractor and cUent of the 
client's status in the study. 

Frequently, applicants interested in classroom training or OJT, including some employer- 
generated referrals, would make their first contact with the provider rather than with the 
college. When this occurred, the contractor would complete au assessment and then refer the 
person to the college for an eligibility interview and random assignment OJT contractors 
prior to the study provided several days of job search activities and sometimes made referrals 
to prospective employers prior to the official determination of eligibility. During the study, this 
was restricted to no more than two days of job search and no direct employer referrals until 
after random assignment was completed and the applicant was identified as in the treatment 
group. 

The intake procedures for a contractor who arranged customfzed training to fill employer 
needs sometimes varied slightly from the procedures described above. In most cases, intake 
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personnel went on-site and, if the applicant appeared to be JTPA-eligible, random assignment 
was conducted prior to obtaining all documentation to support the eligibility. This was done 
to expedite the placement of individuals in these programs. Those who were assigned to the 
treatment group were then required to complete the certification process; any who did not 
meet all eligibility requirements were not enrolled by the SDA. 

Approximately 40 percent of all applicants were recommended for occupational skills 
training over the random assignment phase of the study, with 34 percent of the applicants 
recommended for OJT. Special groups excluded from the study in this SDA were individuals 
sery^ with 3 percent funds or 6 percent funds, which were targeted for hard-to-serve 
individuals (a fifleen-slot program for people with multiple barriers to employment), people 
with handicaps who were to be served by a vocational rehabilitation contractor, individuals in 
a special program for the deaf; and people served through a limited-slot reciprocal agreement 
with the neighboring SDA, 

Adjustments to performance standaxds v;ere not required in either year of the study. 

CORPORATION FOR EMPLOYMENT AND TRAINING. INC (Jersey CItv, New Jersey^ 
Study Context 

The Corporation for Employment and Training (referred to in this report as Jersey City) 
provides JTPA services to a single city area. This urban area, with a population of 224,000, 
is approximately 50 percent black, with the remainder of the residents divided primarily among 
whites, Hispanics, and Middle Eastern and Asian immigrants. With a population density of 
7,000 per square mile and 14.7 percent of the families with incomes below the federal poverty 
level in 1970, the area shares problems similar to those confronting other large urban areas. 

The overall average wage during the year preceding implementation of the study was 
$20,400. This average masks gre^it differences in income within the community, with many 
longtime minority residents earning much less and many recent arrivals in "gentrified" 
neighborhoods working in high-paid jobs elsewhere in the metropolitan area* The 
unemploymeru rate dropped from 10,9 percent in 1984 to 7.9 percent in 1987. However, many 
residents have sought employment outside the SDA because of a severe decline in 
manufacturing and the closing of many factories. The service sector, including Onancial 
institutions and distribution centers, are major employers, but wiage rates are lower than in 
neighboring New York City. 

The Program 

The city contracts with the Corporation for Employment and Training, a private, non- 
profit organization which previously operated the CETA program, to administer the JTPA 
program. The corporation administers the Title IIA 3 percent and 8 percent programs and 
Title in in addition to the 78 j)ercent and 6 percent funds, which are passed through to SDAs 
by the state. 
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The OJT program is implemented by in-house stafiE, who also provide direct placements 
and job searcL The corporation subcontracts for classroom occupational skills training, using 
a total of 23 providers throughout the study period. Contracts ai« performance-based and 
often dui^ted to specific service occupations. Training is provided in areas such as clerical 
tmancial, health and food services, and drafting occupations. One provider received ftinding 
to provide placement services to the Hispanic population during part of the study. Proprietary 
and busm«B schools as weU as vocational technical centers are primary contractore. The SDA 
increased its emphasis on individual referrals and contracts toward the end of the studv 
implementation period. ^ 

In pro^ year 1986, of the U13 terminees, 49 percent were youth and 51 percent 
wereadults. Youth, of whom 22 percent were high school dropouts and 30 percent w^re on 
AFDQ averaged 16 weeks in the program. Adults averaged 14 weeks in the program, and 40 
percent were high school dropouts; 26 percent were on AFDC 

The SDA reported difficult^/ meeting the wage per placement standard for adults and 
recruiUng youth, although in program year 1986 aU performance standards were met or 
exceeded. 

Studv Implementation 

„. ^Applicants' first contact with the program was through the corporation or the contractor 
Ehgi-bdify was determined at the corporation's intake office, and some brief testing was 
conducted at that time. AppUcants who were assessed and determined eligible were called in 
for random assignment at the end of the day. People assigned to the treatment group were 
notified of their status by phone or mail Toward the end of random assignment, procedures 
were modified and random assignment was completed while the appUcant waited so that those 
assigned for services could be immediately referred to an appropriate service provider and 
enrolled by the end of the program year. 

The corporation reported some difficulty recruiting youth throughout the study period 
and some providers reported special recruitment problems. The random assignment ratio was 
adjusted at several points and then briefly suspended in order for the SDA to try to meet goals 
for the program year. Although the SDA met all the performance standards during the first 
year of the study, its performance was slightly down from the previous year, particularly in the 
entered employment rate for adults and youth. A performance adjustment was requested from 
the state, smce this decUne in perf^ormance affected the levels of incentive funds the SDA 
received. A small adjustment was approved. 

Random assignment started in November 1987 and ended in September 1989, with a 
total of 1,686 people in the sample. About 40 percent of the sample were recommended for 
classroom training services, whUe the other 60 percent were divided almost equally between 
OJT and other services. The 3 percent program was excluded from the sample, as were 
homeless people who had to be in a job training program to qualify for shelter. 
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CORPUS CHRISn/NUECES COUNfTV CTexas) 
Study Context 

The Corpus Christi/Nueces County SDA (previously called the Corpus Christi Job 
Training Program) is located in southeastern Texas on the Gulf of Mexico. Corpus Christi, 
with a population of 232,000, has 86 percent of the 270,000 county residents. Hispanics and 
whites have been represented almost equally in the population, at 47 percent each, but the 
Hispanic population is increasing. Black residents represent about 5 percent Approximately 
13 percent of the families in the SDA had incomes below the federal poverty level in 1979. 

The local economy, with many jobs tied to the oil industry as well as tourism, was 
affected by the major recession in Texas in the mid-1980s. The unemployment rate increased 
from 9.1 percent in 1984 to 12 percent in 1987. The overaU wage for the area in program year 
1986 was $18,500. i' b y 



The Program 

At the beginning of program year 1988, the PIC was incorporated and assumed 
responsibility as grant recipient and administrative entity. Staff then became employees of 
the PIC. There are two JTPA ofGces in the SDA, although the site outside the city is not 
a full-service center. The SDA is responsible for Title HA programs, including 3 percent and 
8 percent programs, as well as Titles IB and HI. 

OJT and classroom occupational training are the most frequently emphasized activities, 
with some GED and job search activities also provided. Except for assessment and intake, the 
PIC uses performance-based contracts and subcontracts all services to outside organizations. 
The primary classroom training contractor, a community college, provides classroom training in 
a wide range of occupational areas; JTrA participants are mainstreamed with other students. 
A GED class, however, is offered only to JTPA clients. The Texas Employment Commiision, 
the public employment service, is currently responsible for administering the OJT program, 
although in the first year of the study, a private for-profit contractor fulfilled this function. 
Two community-based organizations are also major contractors to the SDA: LULAC conducts 
job search, job club, and job placement, originally as a stand-alone activity, but more recently 
as a follow-up component for those who have completed classroom training at the community 
college; SER provides concurrent GED and classroom skills training for out-of-school youth. 
Within the last year, the SDA expanded its own services to include some job search activities. 
Reimbursement contracts were used for individual referrals to specific vocational training 
courses, offered primarily through proprietary schools. 

The terminees in program year 1986 were divided almost equally between youth and 
adults. Youth averaged 33 weeks in the program, compared to 25 weeks for adults. Twenty- 
one percent of the youth were high school dropouts and 13 percent were AFDC recipients. 
Eighty-one percent of the youth were age 18 and over. Of the adults terminated, 24 percent 
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were high school dropouts and 16 percent were AFDC recipients. TTie SDA met all of its 
performance standards in program year 1986. 

Study Implementation 

Random assignment began in January 1988 and ended in January 1989, with 1,609 people 
randomly assigned. C enerally, recruitment was not a problem, although specific contractors 
reported some difficulty meeting goals; the SDA felt this was only partly caused by the 
establishment of the control group. During the first year of random assignment, all JTPA- 
fiinded students at the community college were excluded from the sample because enroUment 
in classes had already occurred In the second year, the SDA redefined procedures and only 
new students (or students who had only taken preparatory courses) who were randomly 
assigned to the treatment group were allowed to receive JTPA funding. Excluded fi-om the 
sample were older individuals served through 3 percent funding, the 8 percent program, a small 
number of AFDC recipients in a GED program, and a program funded through Title HA 6 
percent funds that covered work and training-related expenses, such as uniforms and car 
repairs, for AFDC redpients. 

Implementation of the study did require closer coordination between the SDA and its 
contractors, although relationships prior to the study were already good. Responsibility for 
recruitment was shared: The SDA conducted assessments and determined eligibility, and the 
contractors were responsible for additional assessments and random assignment Applicants 
who were recruited and seen first by the contractor were sent to the SDA for eligibility 
determination and referred back to the contractor for random assignment Applicants who 
were recruited by the SDA staff were assessed and screened by the service providere prior to 
random assignment Those determined inappropriate by the service providere were referred 
back to the SDA for further assessment. There were some problems with the flow of people 
and paper as a result of this process. Originally, paperwork was given to the applicant to take 
with him or her to the other agency, but some of it was getting lost Procedures were modified 
so that the contractor picked up a copy of the paperwork and could track who was being 
referred. 

During the sample buildup period, 50 percent of the applicants were recommended for 
classroom occupational training and 42 percent were called in for OJT. The random 
assignment process varied slightly by provider. To expedite employer referrals, the OJT 
contractor completed the call while the client waited. The contractor who focused on out-of- 
school youth scheduled applicants in groups, completed the call-in for random assignment at 
the end of an orientation session, and divided the group to inform them of the results. The 
community college made the random assignment call for all applicants just prior to the start 
of a new semester. 

The SDA did experience some problems with performance over the period of the study. 
During the firet year, funds were slightly underepent and the governor approved an adjustment. 
In the last year, the SDA has requested an adjustment in the post-program follow-up 
employment rate for adults. 
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CRAWFORD/HANCOCK/MARION/WYANDOT COTJNTTES (Marion. HhM 



Stndv Context 



Located in northcentral Ohio, this four-county SDA has a population of 206,000. Marion 
is the largest city, followed by Findlay; each has a population of about 36,000. The population 
is predominantly vvhite. Approximately 7 percent of the families had incomes below the federal 
poverty level in 1979. 

13conomicaIly, there is considerable variation. The county of Hancock has had the lowest 
unemployment rate in Ohio, while the other counties have been adverselj- affected by declines 
in manufacturing. Overall, the unemployment rate for the area, which averaged 10.2 percent 
between 1984 and 1986, dropped to 83 percent in 1987. The average wage in proEram year 
1986 was $18,100. c 6 f e / 



The Program 



The PIC is incorporated and serves as the administrative entity and giant recipient Its 
staff includes employment advisors, testers, and basic education instructors. The central 
administrative ofGce is in Marion. Five JTPA training centers are spread throughout the SDA 
and are staffed by PIC personnel, who conduct extensive assessments and provide case 
management services. Through the training centers, participants can receive basic education 
and GED courses, using computer-assisted instruction. PIC staff also conduct job club and 
arrange work experience and OJT placements. Qassroora occupational skills training is 
arranged on an individual basis through reimbursement contracts with vocational schools and 
technical colleges. 

Participants formerly averaged approximately 40 weeks in the program as they moved 
through a variety of components designed to upgrade then- skills. However, the decrease in 
the unemployment rate, relative to other SDAs in the state, resulted in a reduction in Title 
IIA funds. Program length continued to be reduced during implementation of the study. In 
program year 1986, adults participated for an average of 34 weeks and youth participated for 
an average of 24 weeks. Forty-seven percent of the terminees were adults. Of these, 42 
percent were AFDC recipients and 32 percent were high school dropouts. Nineteen percent 
of all terminees were high school dropouts and 36 percent were ArDC recipients. Seventy- 
one percent of the youth were students and almost 60 percent were 17 years c'.- or younger. 

The SDA met all of its performance standards in program year 1986. 
Study Implementation 

Random assignment began in January 1988 and ended in July 1989, with 1,154 people 
in the sample, of whom more than 1,000 were recommended for services in either classroom 
occupational training (49 percent) or OJT (40 percent). Exempted from the study were older 
individuals served with 3 percent funds and job-ready welfare recipients mandated to participate 
in a job club under a contract with the Department of Human Services in two of the counties 
(at one point, a thi. i county also participated in this project). 
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Because ihe HUe HA funding was reduced, and fewer people could be served, there 
were no recruitment problems during the period of random assignment In the spring of 1989 
ttirolhnents for aU groups except AFDC recipients were fiozen because of the £undin«^ 
shorta^ " 

The need to identify treatment categories prior to random assignment caused the SDA 
to lengthen its assessment process In some cases to ensure that appUcants were assigned to 
the correct group of activiUes. Assessments were individualized, but all appUcants were given 
the same batteiy of tests during half^ assessment sessions. The testing techmaan also 
provided the general orientaUon to the program and briefly eq)Iained the study. After the 
tests and determination of eligibiUty, appUcants met with an employment advisor to make their 
goals final For some people, determination of the appropriate program track was a short 
proc^; for others, it could extend over several days or weeks. Employment advisors would 
tnen discuss theu- cases at a staff meeting so that the final decision could be made about an 
artivity group and appropriate services. Subsequently, an cases handled in that week were 
called in for random assignment AppUcants were notified of their status by phone or letter, 
dependmg on how soon an activity was scheduled to begin. 



EAST CENTRAL IOWA (Cedar Rap ids) 
Study Context 

Hie East Central Iowa SDA includes the ciUes of Cedar Rapids CwpuIaUon 110,000) 
and Iowa City (populaUon 50,000). Total population in the SDA is approrimately 330000 
and outside the two main cities the SDA is very rural The largest SDA office is located in' 
Ctedar Rapids and each of the surrounding five counUes (Benton, Iowa, Johnson, Jones, and 
Washmgton) has a smaller office. About 6 percent of all families in the SDA had incomes 
below the federal poverty level in 1979. 

The local labor market is closely linked to the ..jrtunes of agriculture, through direct 
production and processing of food products. Within the SDA, the University of Iowa is also 
a major employer. Unemployment remained at about 6 percent from 1984 through 1986 then 
dropp^ m 1987 to 43 percent with growth in the local economy. Tht average annual wage 
m the SDA was approximately $17,200 in program year 1986. 

The Program 

The SDA (a private, nonprofit, mulU-jurisdictional agency) administers the JTPA 
program. Each of its six offices is responsible for taking applications, determining eh'^ibility 
and assessing applicants. Staff in the offices are typically experienced, with many having been 
with the agency since the CETA program. 

Oient recruitment was not a problem in this SDA except for occasional problems 
enrolling youth. This was partly because, when the study began, many enrollees were carried 
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into a new program year. In this second program year, funding was not as great as expected, 
so the target number of new enrollments dropped sharply. 

These funding problems had implicaUons for the services provided in the SDA. In past 
years, intensive services such as OJT had been the major activities offered. With the decline 
in funding, SDA managers encouraged staff to provide less intensive services to allow more 
appUcants to be served. Nevertheless, staff continued to recommend intensive services such 
as OJT and classroom occupational training Tor most appUcants. 

The 731 persons terminated from Title HA of JTPA in program year 1986 were equally 
divided between adults and youth. Youth averaged 26 weeks in the program, while adults 
averaged 25 weeks. Seventy percent of youth served were 18 years of age or older, 8 percent 
were dropouts, and 21 percent were receiving ATDC Among adults, 19 percent were high 
school dropouts and 27 percent were receiving AFDC 

In program year 1986, the SDA met all its performance standards. 
Study Implemen->af ion 

Study procedures could be integrated into the normal operations of the SDA with 
lelativefy few changes. In the Cedar Rapids office, appUcau'ons were taken during a group 
orientation session. During tiiis, the staff briefly discussed the study and administered short 
tests of basic skills, i^plicants then met with a coordinator to discuss the program and 
determine any further documentation needed to establish eligibility. Once eligibility vras 
established, appUcants returned for an assessment appointment with a coordinator, during 
which the background information form and informed consent form were completed. On 
completion of assessment, the coordinator recommended individual services and the appropriate 
activity categoty for the study and called MDRC for random assignment 

In the five rural offices, appUcation and eUgibiUty determination were done in an 
individual meeting with cUents. The background information form and informed consent fonn 
were completed during these sessions. TTie assessment process was less formal than in the 
Cedar Rapids office, and its length varied according to the needs and interests of the cUenL 
Once assessment was completed, staff recommended an activity category and called MDRC for 
random assignment 

Random assignment began in Uiis SDA in June 1988 and ended in June 1989. Because 
of the sharp decline in funding for new enrollments during tiie period of random assignment, 
the study sample of 498 fell weU below the original target number. The vast majority of the 
sample were recommended for intensive activities such as classroom occupational training (55 
percent) and OJTs (38 percent). General assistance applicants served by the SDA under a 
state-funded welfare employment program were excluded from the study. 
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HEARTLAND fLalcelaad, FInrirtn) 



Stadv Context 

OkeecSL^Tl^n'Lf^ TZ^ T'^'' ^ ^Sblands. Polk, and 

Okeechobee) m centol Honda between Tampa and Orlando. Seventy-five percent of the 

H^i r^^^T°^ °^ ^^^^'^ (populaUon 50.(^). the largest 

SI.. Winterhaven, a retirement conmiumty. is also toP^^ T^iremaiS 

tZc "S. ^ """'^ developed, though they are dotted with a number of ciUes and 

Sn f^r ^ ^"'"^^ i«^^«y 

to„n„?°^r"^°' ^ ^ "^^^ °f manufacturing, services, and (near Orlando) 

tounsm. The unemployment rate in the SDA stood at 113 percent in promm year 19^ 
dechmng ^dually to 9.4 percent in 1S87. During the Srly 1980s fT^k^J^h 

shanty with the opemng of a major regional shopping center. Average anSuTwag^S 
program year 1986 were $15,200. among the lowest of the SDAs in the stud^ 

The Program 

The Heartland Private Industry CouncQ (PIC), a nonprofit organizaUon, operates the 
P^Srsm vnihm ihc SDA, At the time the study began, clients were s^nS through 
seven offices, three of which were in Polk County. OJTs are arranged by PIC staff but 
da^racm occupatjon^ traning is subcontracted to approximately ten serviti providers in a 
typic^ year, with the Polk County Board of Educadon being the largest Though these service 
?jr^oHH ''^^^ o' "^^PlCstaa-conduct theJTPAintake and eUgibiUty determination. 

Tlie Honda Employment Security Commission (ESQ offices in Lakeland and Winterhaven also 
provide job search assistance under a contract with the PIC and do conduct JTPA intake. 

Qassroom occupauonal training is the predominant service provided by the SDA. In 
program year ly, two-thirds of all adult participants and ahnost 40 percent of youth received 
tos service^ While recniitment had not been a problem in the mid-1980s, finding appropriate 
7^ 2?^ .''PiS^^ demanding courses could be difficult At the time the 

study began, the SDA planned to put somewhat more emphasis on OJT in its service plan. 

In pnagrara j-ear 1986. 3.044 persons were tenninated from Title HA of JTPA: 63 percent 
of them were adulu^ Thirty-five percent of adults were high school dropouts and 15 percent 
were receiving AFDC Among youth tenninees. about 60 percent were 18 years of age or 
older. 11 percent were high school dropouts, and 12 percent were receiving AFDC The 
average participation in the program was 14 weeks for adults and 20 weeks for youth. 

During program year 1986. the SDA met all its performance standards. 
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Stnttv Implementation 



Since 80 percent of JIPA cUents come from Polk Cbunty, implementaU'on of the study 
was limited to that area. The Horida ESC offices in the county were also part of the study. 

Intake and eligibility determination were done in individual meetings witii clients, during 
which the study was explained and tiie background information form and informed consent 
form were completed- FoUowing this, eligible applicants were referred to assessment - an 
extensive array of testing and counseling lasting up to six hours and cuhninating in tiie 
development of an employment development plan. Staff then designated an activity category 
for Uie study and reviewed witii tiie applicant once again tiie informed consent form. As each 
assessment was completed, staff called MDRC for random assignment and informed tiie cUent 
of tiie results. Special exclusions from the study included a limited number of extreme hardship 
cases and appUcants who had completed a state-funded remedial education program and 
returned to JTPA for fiirther training. 

The SDA began random assignment in May 1988. By October 1988, when only about 
500 persons had been randomly assigned, staff were concerned about tiie dramatic drop in 
applicants and tfie shortfall in enroUments. Hiis was occurring because of a decline in tiie 
unemployment rate in tiie SDA, tiie opemng of the major retail center, and problems recruiting 
cUents because of pubUcity about funding cuts in tiie SDA for JTPA services. AU of these 
factors combined to present tiie SDA witii a new challenge requiring more intensive recruiting 
and better retention of appUcants and new procedures to address tiie needs of hard-to-serve 
clients. Despite additional techm'cal assistance on ch'ent recruiting and retention, tiie SDA 
coritinued to experience a major enrollment shorti'alL After im'tially suspending random 
assignment for two months, the SDA decided to end it in January 1989. 



JOB TRAINING OF GREATER OMAHA (Nebraska^ 
Study Context 

Located in eastern Nebraska, this SDA serves approximately 500,000 people and includes 
the greater Omaha metropoUtan area. The largest county is Douglas, witii a population of 
398,000, of whom 312,000 live within tiie Omaha city Urm'ts. The SDA also includes Sarpy 
(population 86,000) and Washington (population 16,000) Cbunti'es. The majority of the 
population is white, with black residents making up the largest minority group. In 1979, 6.6 
percent of all families had incomes below the federal poverty level. 

The employment base includes large manufacturing and transportation employer:; as well 
as jobs in clerical and administrative work and sales and sales-related employment. The 
Strategic Air Command (SJi.C) is the largest employer, followed by other large service-sector 
firms. 
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TTie unemplqyment rate ranged between 4.7 percent and 5.6 percent in the four years 
preceding the study and was 5 percent in 1987. Hie average annual wage was $17,700 in 
program year 1986. 

The Program 

Job Training of Greater Omaha (the new name for this SDA) provides JTPA services 
in the three-county area; it was formerly caUed the Omaha OfBce of Employment Resources. 
The city of Omaha is the administrative entity and grant recipient, but an active PIC guides 
the program and is closely invoked in major decisions. The SDA also administers Htie ffl. 
Hght percent funds are used to enroU people in basic education classes combined witii work 
otpenence; tiiese are usually foUowed by OJT or classroom training. Hght percent funds are 
also used for occupational training and other classes at the local community coUege. 

Qassroom occupational skills training is the predominant activity for Htie IIA JIPA 
participants. During implementation of tiie study, most classroom training participants were 
ti^med tiirough five major contractors, predominantly community-based organizations. Qasses 
provided training in service occupations, such as healtii services, cooking, finance, office 
equipment repau-, and clerical work. Contracts were performance-based. Job search is 
conducted by in-house stag; v*o also mom'tor and arrange OJT placements. 

In program year 1986, 55 percent of tiie 853 Tide HA terminees were adults and 45 
percent were youth. Youth averaged 14 weeks in tiie program and 14 percent of them were 
high school dropouts. Sixty-four percent of tiie youtii were age 18 or over. Adults averaged 
10 weeks m the program and 17 percent were high school dropouts. AFDC recipients 
represented tiie same proportion - 34 percent - of tiie adult and youth terminees. " 

The SDA met all performance standards in program year 1986, although in tiie year 
before tiie study it underspent its allocation for youth. 

Study Implementation 

The service providers were responsible for recruitment, a function that had been assigned 
to tiiem tiie year before tiie study began. However, tiie SDA also provided general recruitment 
for the program as a whole. 

Intake, consisting of completion of an appUcation and eligibility determination, was 
normally performed at tiie central Omaha office. (The office moved to a new location in 
early 1989.) During the intake interview, tiie background information form was filled out by 
staff and the study was briefly explained. Eligible applicants were then referred to an 
assessment and testing workshop, also conducted by SDA staft At the workshop, the 
\ideotaped explanation of the study was shown and the informed consent form was signed. 
Staff ipcctings for groups completing the assessment and testing workshop were used to 
approve counselors' recommendations of activities. 

People recommended for classroom training were referred to the appropriate contractor 
for a provider's assessment. People determined appropriate by the contractors were then 
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scheduled for an empIoyabUity development plan (EDP) interview with SDA staff; which could 
take place in-person or phone. After the EDP interview, SDA staff completed random 
assigimient and notified the contractor and applicant of his or her status* People recommended 
for OJT were scheduled to meet with an SDA job developer for an EDP interview immediately 
following the staff meetings discussed above. Random assignment took place after the plan was 
made final 

To implement the study, several changes were made in the assessment and testing 
workshop. Previously, the workshop had extended over two days and was followed, for some 
applicants, by a two-day job search assistance workshop. However, the SDA had difficulty 
retaining people through this whole process and requested on-site training by Clygnet Associates 
prior to the start of the study. As a result of the training, the workshop was shortened to a 
half-day. The SDA also sought to discourage contractors from conducting an additional 
assessment, although some continued to require their own job-readiness assessment 

G^nrenty-three percent of the people randomly assigned were recommended for services 
in classroom occupational skills training. OJl' was the second most frequently recommended 
component, accounting for about 22 percent of the sample. 

Random assignment began in October 1988 and continued through September 1989, 
with 1362 people randomly assigned In program year 1988, the SDA did not meet the adult 
entered eniployment rate standard, and the director did not request an adjustment In program 
year 1989, the SDA received adjustments in its adult and welfare entered employment rate 
standards. The original sample goal of 1,600 was not met because two months before the study 
ended, the SDA was authorized to exempt from random assignment applicants recruited and 
referred for intake by the contractors. 



LAND OF THE OZARKS fSpringfield. Missouri) 
Study Context 

The Land of the Ozarks SDA (referred to in this report as Springfield, Missouri), located 
in southwestern Missouri, senses seven counties: C3iristian, Dallas, Greene, Polk, Stone, Tanej , 
and Webster. Springfield (population 133,000), in Greene County, is the largest city and the 
location of the SDA's central office. The SDA is primarily rural, with a total population of 
304,000, of whom more than 90 percent are white. An estimated 10 percent of the families 
had incomes below the federal poverty level in 1979. 

The economy has been strong, with employment increasing in both the service sector, 
which employs the majority of the labor force, and manufacturing. Greene County also has 
a strong agribusiness base. The unemployment rate has gradually declined in recent years, 
from 7.7 percent in 1984 to 5.7 percent in 1987; in areas that are strongly influenced by 
tourism, seasonal changes can bring the rate as high as 20 percent The average wage in 
program year 1986 was $15,200. 



TTie Program 

Hie City of Springfield Human P^sources Department is the administrative entity and 
grant recipient JTPA services are provided through the Job Council of the Ozarks, with fuU- 
service offices in Springfield, Branson, and Bolivar, a regularly staffed outreach office in 
Buffalo, and applicaUon sites in Kimberling City and MarshfielA Staff in each office' are 
responsible for intake, assessment, service delivery, and implementation of the study procedures. 
The PIC and local elected officials act as policymakers and nlannere and provide Drogram 
oversight r r e 

Most of the occupational skills training programs are located in the Springfield area and 
transportation is a problem for many SDA residents, so there is heavy reliance on OJT 
programs to meet the population's training needs. Qassroom ti-aining was provided primarily 
m health occupations, with programs such as licensed practical nursing and respiratoiy therapy. 
Welilmg, office occupations, and auto mechanics were also among the offerings. GED training 
was provided to dropout youUi through a reunbursement contract vwth a commum'ty-based 
organization. Youth also received pre-employment skills training and job placement services. 
Work experience and a job-seekers' clinic were provided for both adults and youth. 

In program year 1986. 1,117 people were terminated fi-om the program. Sixty percent 
of the termmees were adults and 40 percent were youth. Youth partidpated for an average 
of 20 weeks. Eleven percent were dropouts and 19 percent were AFDC recipients. Sixty- 
three percent of the youth were age 18 or older. Adults partidpated an average of 22 weeks. 
Twenty percent were dropouts and 14 percent were AFDC recipients. 

The Job Council of tiie Ozarks has consistently had high outcomes on performance 
standards, which have placed it between first and third among the SDAs in Missouri. However, 
in program year 1986, the average wage for adults employed fell slightly below the model- 
adjusted standard. 

Study Implementation 

The SDA made significant changes in its recruitment, intake, and assessment process 
midway through the implementation of random assignment Using the training provided by 
Cygnet Assodates, it streamlined its enrollment process, eliminating an initial screening 
interview and some tests and emphasizing program benefits. SDA staff developed a variety of 
materials to explain and facilitate implementation of the study. 

In the Springfield office, applicants were scheduled for motivational group orientations, 
dunng which the study was explained, the informed consent form signed, and the background 
information form (BIF) handed out with a list of other documents the applicant would need 
for the eligibility interview. Applicants were then scheduled to see a technician, who 
determined eligibility, briefly assessed the applicant to determine appropriate activities, and 
completed the BIF. While the applicant waited, a derk made the random assignment call and 
the applicant was informed of the results. People assigned to the treatment group were then 
referred to the next step in the service plan, which was agreed to during the assessment 
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Procedures for the other, smaller offices did not include the group orientation- 
Applicants were informed in-person or by phone about the study and the paperwork needed 
to determine eligibility. Whenever feasible, the eUgibility interview and assessment and random 
assignment phone call occurred during a single visit 

Recruitment was difBcult owing to the low unemployment rate, and staff reported that 
they did relax some of their entry requirements during the study. Because of various concerns, 
a decision was made to shorten the length of random assignment ftom 17 12 months. The 
study did affect the SDA's ability to fiiUy expend Title HA funds, and the SDA said that 
performance declined, although performance standards v/ere met. 

Over the study period, about 65 percent of those randomly assigned were recommended 
for an OJT; about 17 percent were recommended for classroom skills training; and 18 percent 
were recommended for other services. The 3 percent program, administered outside the SDA, 
was excluded from the study, as was the licensed practical nurse traimng. The research design 
was modified to enable the SDA to add a short OJT component for some classroom training 
participants. 

Random assignment began in April 1988 and continued through March 1989. A total 
of 1,202 people were assigned for the study sample. 



LARIMER COUNTY fFort Collijs. Colorado^ 
Study Confeirt 

Located m the northcentral part of the state, approximately 65 miles north of Denver, 
this rural county had a total population of 150,000 in 1980, 65 percent of whom Uved in Fort 
Collins or Ix)veland, site of the two JTPA offices. By the mid-1980s, the SDA's population 
was approximately 200,000. The population is predominantly white (92 percent in 1980); the 
largest minority is Hispanic (6 percent). Only 5.9 percent of the families were classified as 
having incomes below the federal poverty level in 1979. 

The county experienced strong growth in the late 1970s and early 1980s, but in the 
years immediately precedmg the study, unemployment began to rise - from 4.8 percent in 1984 
to 7.4 percent in 1987. The service sector employs the largest number of workers, followed 
by trade and manufacturing. Colorado State University is the largest employer, with Hewlett 
Packard and Anheuser-Basch next in number of employees. The overall average wage in 
Larimer County was $17,'"00 in program year 1986. 

The Program 

The county government is the planning and administrative entity and grant recipient for 
JTPA SDA staff are county employees in a department called Larimer County Employment 
and Training Services (ETS), but an active Private Industry Council guides the program. The 
SDA also administers the Title IL\ 3 percent program and Title IIL Fort Collins is the 
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administrative headquarters and serves about 60 percent of the SDA's clients, with the 
Loveland office serving the remainder. When the study started, the SDA and PIC also oversaw 
the local employment service activities and stag; which are co-located with the SDA offices. 
But acUons at the state level gradually resulted in a return to the more traditional, separate 
administrative strueiure. 

ETS, because of its relatively small Tide DA 78 percent allocation, serves the majority 
of Its clients through short-term, low-cost, but highly structured activiUes, many conducted by 
Its own personnel: job club, a three-week prograrfi with its own training area and phone bank, 
IS offered monthly in each office; a one-week assessment program and youth competency 
workshops are also ongoing. A workshop to teach job retention skills was recently added. 

Other programs are contracted to schools on a cost-reimbursement basis. Individuals 
recommended for classroom occupational skills training begin their time in training with a 
two-week Mir^r planning program for JTPA participants conducted at Front Range Commum'ty 
CbUege. ETS staff contribute to curriculum development and consult with instructors and 
chents in these programs. The high level of interaction between staff and otiier agencies is 
also evidenced by a formal network of 19 community organizations, including groups not 
involved with JTPA, who meet regularly to foster coordmation and develop solutions for 
individual problem cases. The SDA has received national recognition for its networking system. 

ETS, through the PIC, has also established itself as an employer resource in two key 
ways. First, several major companies have used ETS and tiie employment service as a center 
for screemng and testing prospective employees. Whfle botii JTPA-eUgible people and others 
who are not disadvantaged have been placed through this service, few of Uie large employers 
use the OJT program. Second, as a service for employers, the PIC annuaUy sponsors a 
Busmess Expo, bringing in experts to educate area employers. 

In program year 1986. Larimer County ETS terminated 401 individuals from Title EA 
of JTPA The average time between program enrollment and termination was 26 weeks for 
youth and 22 weeks for adults. The SDA did not meet its average vrage at placement 
benchmark in program year 1986, but the cost per adult placement was far lower than 
expected 

Study Implementation 

Before the study was implemented in December 1987, enrolhnent occurred prior to 
assessment in order to discourage the development of a screening process tiiat might eliminate 
the harder-to-serve. To accommodate tiie research design, the SDA decided to implement a 
mini-assessment (locally called a pre-EDP) immediately foUowing eligibility determination so 
that applicants could be recommended for a treatment category prior to random assignment 
ETS worked with a consultant to develop a checklist that was used to assist staff in determining 
an applicant's appropriateness for each of the three treatment categories. 

Following the pre-EDP, the random assignment call was placed and staff informed people 
of their research status in person. People assigned to the treatment group received additional 
assessment either on the same day or shortly thereafter. Because of the large number of 
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participants served in job search and assessment activities, the "other services" category for this 
SDA was set at the comparatively high level of 70 percent During random assignment, 64 
percent of the applicants were recommended for tWs treatment group. Twenty percent of 
the people randomly assigned were recommended for classroom occupational skills training, with 
only 7 percent of the people randomly assigned recommended for the OJT services category. 

To avoid jeopardizing ETS's good relationship with other agencies, ETS and the 
researchers conducted briefings on the study procedures for community agencies prior to the 
start-up of random assignment Groups not included in the research in this SDA were all 
individuals 55 and over and referrals for service j&rom Project Self-SufiBciency (PSS). PSS is 
a federally funded demonstration project designed to mcrease the self-sufficiency of AEDC 
recipients. It uses the resources of multiple agencies to provide comprehensive services - 
including housing, employment, and training services - to help people make the transition from 
welfare to work. 

Midway through random assigninent, for about six months, the SDA also conducted 
group orientations prior to the eligibility interview. During these motivational sessions, a 
localized version of the random assignment videotape was shown. Recruitment activities did 
not change substantially during the sample buildup period, and particularly in the Loveland 
office enroUment goals were difficult to meet 

Random assignment ended in Larimer County on September 30, 1989, with 1,027 people 
in the sample. 



MACON/DE WITT COUNTIES fPecflfnn niinols> 
Stud? Context 

The Macon/De Witt Counties SDA, located in central Illinois east of Springfield, serves 
the two counties. They have a total population of nearly 150,000. Decatur, a city of 
approximately 90,000 locate to Macon County, is the home of the larger JTPA office; a 
second office is located m De Wtt County in the town of Qinton, which has about 8,000 
residents. Outside of Decatur, the SDA is primarily rural, with scattered small towns. Decatur 
has a significant black population, but the remaming parts of the SDA are primarily white. In 
1979, 7.8 percent of the fanulies in the SDA had tocomcs below the federal poverty level. 

The Macon/De Witt labor market continued to experience high unemployment 
throughout the 1980s, despite the national economic recovery. The unemployment rate was 
10.7 percent in 1984, 10.2 percent in 1985, and 113 percent L 1986; it declined slightly to 10.6 
percent in 1987. Plant closings or layofEs in several large heavy manufacturing firms occurred 
in the 1980s, and these cuts have caused a leveling off in wholesale trade, finance-insurance, 
and real estate employment Construction of a nuclear power plant in Qinton did increase the 
demand for labor in that part of the SDA. The average annual wage in the SDA was $22,300 
in program year 1986. 



-173- 

20; 



The Program 



The JITA grant recipient is Macon County, which administers the program. An 
- t^^?^J^ T P""^^ assistant director haSng served 

^ °^ T"^ P'?^ ^ ^ I>««tur and Clinton conduct 

^TsS^^^A T^'* ^'"^'y- ^ "S^^ unemployment rate in 

h^^^of^ ° -T^ '° actively.recruit cUents; 40 percent of appUcants L walk-ins, and 
the bulk of the remamder are referrals from public aid or the employment senice. 

The primary seiyices offered by the SDA arc OJT and job search assistance though 
^rSTir?! "^ri ^^'^ -^^^ P'^«» to increase Ts^Sg 

2e On Jl ^ '^u ^- ^'"^ « "^^^^ coUege opera^ 

wWc?^e 4a n'T^ '^''2°f '^'^ ^^^^ "°der an arraigei^nt by 

uStS to J '^-^^ °f °f staff, with the payment 

Jr^ntlu^' '^Z^^^^f^'r'''^' ^^°"Sh the coUege received bonuses when ft pS 
certam ^oups m jobs. The SDA also funds basic education (through a regional school dS 

totoroS^"'"'"' ' °^ ^'^tilities Letimes se^Hs a S 

Of 621 clients terminated from the SDA's Title HA program in program year 1986 58 
p^nt were adults and 42 percent were youth. Adults averaged 29 wee^of Smfn't 2 
JIPA. while youth averaged 20 weeks. Among adults, the majority - 63 percent - were male 
bS^'fTri? °^ ''^"I'' f'i^,.^''^^' °f y°"th were white; most of he remainder were 

«dut .T^nl ^[""f ^'^^ '^^P^"*^- Twenty-two percent of 

adults and 30 percent of youth were receiving AFDC on entiy into the program. 

placem^r^" ""^^ ^ performance standards except average wage at 

Study Implgmentation 

The study procedures could be introduced in this SDA with relatively few ch-nges in 
normal operations, partly because the existing high demand for the program meant that demand 
for program services clearly exceeded avaUable program slots. When clients initially inquired 
about the prograi^ staff provided them with information on program services and elidbility 
rules and scheduled an mdmdual intake interview with a counselor. At that appointment, the 
counselor explained the study, provided an additional orientation to the prograrTciled oiit an 
apphcauon and background information form with the client, and began the assessment process 

for "^nnLV^u D T'^'^" "^'^ '"P""^' 'o agree on an appropriate service plan 
for apphcante. Random assigriment calls were made following these meetings. Seventy-s^en 
percent ot those randomly assigned were recommended for OJTs. 

Only one special change was made in the usual procedures to implement the study. 
iT.r n.t^ u ""J^"" assignment, SDA staff had referred applicants recommended for 
^° search assistance to the local community college providing these services 
without designaung which service should be provided Prior to the study, SDA staff were 
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considering designating a single service at referral and used the start of random assignment 
(with its requirement of a single treatment category) to introduce this change. The only group 
specially excluded from the study in this SDA was older workers recommended for a special 
OJT program. 

Random assignment began in November 1987 and continued through July 1988, with 
the total sample reaching 471, below the original target number. The SDA chose to end 
random assignment early in order to participate in a state-initiated economic development 
project, which required them to refer large numbers of carefully screened job candidates for 
possible employment at a new plant in the area. 



NORTHEAST INDIANA fFort Wavne) 
Study Context 

The Fort Wayne Area Job Training and Development Corporation or Job Works, located 
in Indiana's northeastern comer, serves eight counties: Adams, Allen, De Kalb, Huntington, 
Noble, Steuben, Wells, and Whitl^. (Northeast Indiana is the SDA name and the name used 
in this report) Fort Wayne, a city of approximately 175>000 in Allen County, is the largest 
urban center in the SDA and the site of the SDA's central ofGce. Outside of Allen County, 
the SDA is primarily rural, with many small towns. The SDA's total population is 
approximately 500,000, about 93 percent of whom are white, with black residents making up 
the bulk of the remaming population. An estimated 6 percent of the families in the Fort 
Wayne area had incomes below the federal poverty level in 1979. 

A stroug recovery from the recession of the early 1980s dropped the unemployment 
rate from 8.9 percent in 1984 to 5*1 percent in 1987, with some of the rural counties having 
unemployment rates well below this level During 1988 and 1989, employers were having 
difficulties filling jobs, and some were seeking workers from outside the area. Manufacturing 
remains the largest employer in the SDA, providing jobs for one-third ot all workers in 1987. 
Between 1983 and mid49S7, manufacturing employment increasec y 26 percent or 16,500 jobs, 
though many of the new manufacturing jobs pay less than heavy industry jobs lost in the late 
1970s and early 1980s. The average wage in the Fort Wayne area was $17,800 in program year 
1986. 

The Program 

The Private Industry Council is an incorporated nonprofit organization, which has an 
agreement with Job Works to administer the JTPA program through an office in each of the 
eight counties in the SDA. Each uobWorks office is responsible for its own intake, assessment, 
service delivery, and - during th^ period of the study - random assigiunent A recent stale 
iiutiative to merge the actrvities of the Indiana Employment Security Division (the state 
employment service) and JTPA has led to greater cooperation between these two agencies at 
the local level and co-location of offices in many of the rural counties surrounding Fort Wayne. 
Even before this merger, most JobWorks applicants had already registered with the 
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j^^c^^ Security Division, and those vAio had not y^tt jequired to register as part of 

offi^^*^°i?° °^ P^Sr^ «-'ttiphasis vaiyine by local 

office dependmg on the avaaability of service providers. TiS primJy servi^fe^^ 

Sif S^' TT"^ f*' : ia the areas close to Fort Wajle -Tb^r^m o^pSoS 

Sl«„ programs are run by service providers, most of^om operarS 

performance-ba«d contracts. Hie site subcontracts vnth about 20 service provi<ter^ ^rSe 
jmjal number of cUents ranging from 5 to 40 per program. Service^SEtduTe Ic^ 
technical mstitut« and schools, commurity-based orgadzauons. and o^TaZZs^t 

were Zil^^^dantl^r^T ^""^ ^ ^"^^ P^'Sram year 1986. about two-thirds 

70 LrSit wh ^ .^'"n?' ""T °^ ""^"^r'' °^ or older. Approximately 

/u percent were white, 27 percent were b ack. and 3 percent were Hisnanic <:«,pnt^«« 
percent of the adults and 13 percent of the youA served rSlS(?Se 31 S^nTS 

v^^ ^ school dropouts. Adulu se^^ avS 13 

H-seks m the program, ^e youth averaged 22 weeks. «ivcrdgeG is 

JobWorks met aU of its performance standards in program year 1986. 
Study ImpIemcHtfltinn 

occurrS'aS^'S.rT'^t TT' '"^"^ ?°? ^^'^^ ^ g^°^^^y «°dom assigmnent 

oca^edafter the chent had completed aU appUcation forms, been determined eligwTfor 
^t^^ .assessed by JobWorks stafi; and been recommended for specific servicS "Ae 
background mformaUon form for the study was completed along wShother a^Ucation 

^ ''""^^ ^ assessment interview. In the Fort Wayne office, assessment 

2f tLTo^TlvTr'^^^^- ? ^ accommodate the study dSgn, JobSS 

Service recommendaUons v.ere made at the weekly staff mer' es in the laropr nmr« 
on an individual basis in tbe smaUer offices. Fifty^ght percent ofaU ^^i' LdT^ 
assigned were recommended for OJTs, 33 percent for otheV services, and 10 percent for 
dassroom occupaUonal training. Random assi^ent calls were made on a baL^^l a 
scheduled times, though individual calls at other times were also possible. 

becau^S fhe ?T °^ '1^°"^ ^if^^nu client recruitment became more of a problem 
^use of the dechne m the unemployment rate. The SDA made- extensive changes in 
emiitment materials and introduced new procedures to retain a higher percent of apXnl^ 

ml m>T'H • r^'"^ ^""'^^ ^ ^^^^ welfarrrecipientsT Tead 

into JTPA and a second for hard-to-serve youth) experienced special recruitment problems and 
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were exsir>-t from the study. Staff did notice that over time they were working with less 
skilled clii'^^.'^. 



Random assignment began in this SDA in November 1987 and continued through August 
1989, reaching a total sample of 3,608. 



NORTHWEST MINNESOTA PRTVATF iKHTT STRY COUNCIL (^Crookstoa and Thief River 
Falls) 

Study Context 

Zbe Northwest Minnesota Private Industry Cbuncfl (PIC) serves seven couKfies (Kittson, 
Manhall, Norman, Pennington, Polk, Red Lake, and Roseau), which comprise a very rural area 
in the comer of the state abutting North Dakota and Canada. The population of the entire 
SDA is about 100,000, and its three largest towns (Crookston, East Grand Forks, and Thief 
River Falls) each has slightly under 10,000 residents. The population is primarily white, and 
about 11 percent of all famiUes had incomes imder the federal poverty level in 1979. 

The local ax)nomy is agriculture-based, with fiarming, food processing, and related trade 
being the major industries. The severe winters make for a short growing season, and sugar 
beets, potatoes, and wheat are the primary crops. The unemployment rate in the SDA stood 
at 103 percent in 1984 and remained above 9 percent until 1987, when it dropped to 8.6 
percent The average annual wage of $13,600 was the lowest of any SDA in tiie study. 

The Program 

The PIC is an incorporated nonprofit organization that is the JTPA grant redpienL It 
does some ch'ent recruitinenti but the high unemployment rate has allowed the JTPA program 
to rely primarily on walk-ins and referrals from other social service and education agencies. 
The Minnesota Job Service operates most of the JTPA program in Uie SDA under a 
subcontract from the PIC In offices in Crookston and Thief River Falls, Job Service staff 
offer the usual state employment service job listings plus JTPA and other state-funded 
programs for welfare redpients. The local community action agency conducts intake for a small 
older workers' program which provides work experience and job search assistance. 

Job Service staff directly provide job search assistance, career exploration, OJT, and 
work experience. Thqr refer people interested in classroom occupational training to the local 
community college, techm'cal institute, or campus of the state university. Those seeking adult 
basic education and a GEO are referred to a local adult learning center for services. A local 
technical institute also provides an intensive job search assistance course, primarily for those 
participating in classroom occupational training. OJT and classroom occupational training are 
the two most common Titie HA activities. 

In program year 1986, 408 people were terminated from Title IIA, with adults making 
up 57 percent of all terminecs. High school dropouts made up about 15 percent of adult 
terminees, as did AFDC redpients. Among youth, dropouts were only 2 percent of those 
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Sfndv Imp leraeBtflHnn 

th. J^n ^""'"^ ^ ^"^"^ ^^"^ ^ Orientation to 

& itTSJ. ^"^f^ ^'^L"- ^^^^ ^ ^^^"^^ J^A and interest^ 

^ appKcaUon. Job Service staff then determined 
ehgibibty, conducted an assessment of needs, and designated appropriate servi^ 

h«rt«i!?5 P'P^^f^ ^« integrated into the usual Job Service intake by completion of the 

sS comn^tS^L^f ' paperwork was done in a group, while the Crookston 
staff completed study forms during an individual meeting ivith clients. CSioice. of Z 
appropriate activity category was typically made based on an individual meeting with the client 
dwmg which an assessment of needs, interests, and current skills was made. Staff then caUed 
MDKC for random assignment and informed the cUent of the result and nest steps. 

fiind JltiS"^ ^^'^ ^ ^ '^^^ ^ implementing the study concerned a state- 

"to ^^r^T "^^ ^"^'^ ^ '^^"tractua! obUgaUon 

to use JTPA funds to serve welfare recipients referred by the county welfare departments 
These referrals were exluded from random assignment and were not a part of the sm^ 

Random assigmnent began in August 1988 and ended in May 1989. The SDA sUehtk 
acceded Its ^et sample of 550. Virtually aU those randomly assigned had 

OAKLAND fCalifornffl) 
Stndv Cnnferf 

. p f a population of 340.000, is one of 49 service deliveiy areas 

m phform^ Its populaUon is approximateiy 40 percent white, 30 percent black, 25 percent 
Asian, and 5 percent other minorities. There are 6300 residents per square mfle. and 87 
percent of the famihes had incomes below the federal poverty level in 1979. 

f,.,i.i,^'^°' ^ '^"'^"^ transportation (including shipping), manufacturing, government, and 
health ^re. Large employers are Kaiser, Pacific BeU. Merritt.Peralta Medical Supplies Civic 
Gas and Electnc. and Childrens Hospital The unemployment rate fiuctuated in the year, 
preceding the study miplementaUon: It was 7.0 percent in 1984. 6.8 percent in 1985 8 7 
percent m 1986. and 7.7 percent in 1987. TTie overaU average wage in the city was 52ioOO 
in program year 1986. & / « *^wu 
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The Pro^m 



The Private Lidustiy Council (PIC) is incorporated and shares admimstrative rcsponsibiUty 
for the program with the Gty of Oakland. The city is also the grant redpient and planning 
entity for the program. The PIC and city have defined their oversight and admimstrative 
functions so that the city is responsible for determining applicants* eli^ility for programs and 
for tracking participation in the SDA's programs, while the PIC awards and monitore contracts 
for services* 

A subsidiary of the PIC - the Oakland Private Sector Corporation (OPSQ - provides 
direct chent services. The SDA also contracts with numerous outside agencies to provide 
services to the eligible population. Outside of OPSQ contracts tend to be targeted to serve 
particular subgroups or to provide training through one of three activity categories defined for 
the research. All subcontracts are performance-based. 

Classroom occupational skills traimng is the predominant activity. Seven of the 
contractors (commmiity-based, private for-profit, and non-profit organizations) provided 
classroom traimng. Two of these also had contracts to provide OJT, along with another 
contractor who was also funded to develop OJTs, primarily with the Korean population. Two 
other contractors provided job search services, with one of them targeting services to people 
55 and over and the other enrolling veterans. 

OPSQ, the largest contractor, provides classroom training, OJT, and other services to 
all JTPA-eligibles and conducts an older workers' program witii Title HA 3 percent and 78 
percent funds. At the same time the study began, a new program, funded tiirough Utie HA 
6 percent fimds, was initia te to provide training in non-traditional jobs. The Center for 
Employment and Training (CET), the second largest contractor, conducts training programs in 
four occupational areas - office occupations, shippmg and receiving, auto mecham'cs, and 
maintenance - and provides a small number of OJTs. 

In program year 1986, there were 934 people terminated from Titie HA Sfacty-one 
percent were adults and 39 percent were youtL Youth, 40 percent of them AEDC redpients 
and 4 percent of them dropouts, averaged 11 weeks in the program. Fifty-four percent were 
18 or older and 56 percent were students. Adults averaged 12 weeks in the program. 
Fourteen percent were dropouts and 15 percent were AFDC redpients. 

In program year 1986, the SDA met all its performance standards. 
Study Implementation 

The procedures for the study were integrated into the normal operations of the SDA 
with relatively few changes in the flow. However, the involvement of multiple organizations, 
providing different services to different subgroups, made the process of adapting the study to 
the site complex. Because of multiple population subgroups with varying degrees of proficiency 
in English, the agreement to participate form was translated into nine languages. In addition, 
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the cxcJysion of several programs and all youth from the study sample compKcated the initial 
screening process. 



Youth were excluded from the study because of recruitment problems with this group 
A residential treatment program for court-referred offenden; was excluded. Also excluded 
was a basic education program funded through Title DA 8 percent funds. These funds were 
earned over from the previous year and then targeted to serve AEDC redpients in a state 
weifare-to-wt^rk iniUative. Also exempted from the study were a program provided through 
an a^ment xvith the Department of VocaUonal RehabiUtaUon to place people with handicap 
in jobs and a program for homeless individuals (paid for with 6 percent fimds). 

Although contractors were responsible for recruitment for their own programs, people 
also apphed to JTPA by going to the SDAs central intake unit In order to identify fiie 
correct treatment categories for appUcants, providers were dhdded into two groups: those that 
offered only one major category of service (classroom training, OJT, or job search) and those 
that could offer mulUple types of services. The first group was labeled "single service 
providers and the second group, represented by CET and OPSC, "mulUple service providers." 
me foUowmg descripdon of the flow is the same for both groups, except that CET and OPSC 
rompleted most of their assessment, recommended people for activity categories, and placed 
the random assignment phone call themselves after eUgibility had been determimL 

In most cases, cUents appUed for programs directly at the service provider. In these 
instances, contractors conducted program orientaUons and assessments, which varied in length 
and mtensity. During the im-Ual meeting(s), the study was explained and the background 
mfomauon form (BIF) was given to the appUcant along with other intake documents needed 
for the eligibffity interview. Contractors, particularly those that served predominantly limited- 
English-speaking cUents, sometimes helped appUcants complete the BIF. EUgibility interviews 
with city staff were then scheduled at either the central intake unit or the service provider site 
The BIF was flnalized, the agreement to participate form signed, and the raudom assignment 
phone call made during this interview, assuming aU supporting documentation was avaUable. 

As noted above, some cUents appUed directly to the JTPA office. AppUcants whose 
imtial pomt of contact was the central intake mut had their eligibUity determined and the study 
explained. Then they signed the BIF and agreement to participate form. After a brief 
assessment of mterest, the appropriate activity category was identified and random assignment 
took place, foUowed by the referral of those in the treatment group to the appropriate 
provider. rr t- 

Random assignment began in July 1988 and ended in March 1989, with 1,072 people 
randomly assigned, slightly more than the targeted sample. Qassroom training in occupational 
skills was the predominant activity, with 50 percent of the sample recommended for this activity 
group; 8 percent were recommended for OJT. 
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FROVIDENCEyCRANSTON (Rhode Island) 



. Study Context 



Hie Providence/Cranston SDA serves these two adjacent, older urban areas in northeast 
Rhode Island. Providence, with a population of approximatefy 155,000, is the state capital and 
largest dty in the state, while Cranston has approximately 70,0(K) residents. About 8 percent 
of the residents of the SDA are blacks, and all minority groups comprise 13 percent of the 
total population. Nine percent of families in the SDA had incomes below the poverty level 
in 1979. 



The Providence/Cranston area has historically been one of the most industrialized in 
the countT', with a higher than usual percent of employment in manufacturing. In recent 
years, several of the state's major employers (makers of jewelry and sihrerware and the U.S. 
Navy) have cut back emplqyraent, but other sectors (including services) have growa with the 
New England recovery of the early and mid-1980s. Unemployment in the SDA stood at 12 
percent in 1984, but dropped to 43 percent in 1987 as the SDA beneGted from the booming 
regional high tech and services economy. The average annual wage in the SDA in program 
year 1986 was $17,000. 



The Program 



The Providence/Cranston Job Trammg Partnership (PCJTP) is the grant recipient and 
administers the program through offices in each of the two cities in the SDA, Recruitment 
of cUents, done by both the PCXI? and service providers, has become increasingly a challenge 
as the area's unemployment rate dropped throughout the mid-1980s. Intake, eUgibiUty 
determination, assessment, ard counseling are performed by the SDA staff in each office, 
though the Cranston office must file case documentation of eh'gibility in the main Providence 
office before assessment can be scheduled. 

PCnP staff arrange OJTs for clients, but the remainder of SDA services are provided 
by subcontractors operating undei a mix of performance-based and unit-cost contracts. Among 
the services provided in this way have been adult basic education, English as a second language, 
classroom occupational training, vocational exploration programs (pre-employment skills and 
work experience for youth), and job clubs. 

Of tiie 639 persons terminated from Title IIA activities in program year 1986, 54 percent 
were adults and 46 percent were youth. High school dropouts comprise 27 percent of adult 
terminees, and AEDC recipients were 35 percent of terminees. For adults, the average 
enrollment in the program was 12 wet 5. The youth served in this SDA are among the mc^t 
disadvantaged in the study: 21 percent are high school dropouts (the second highest percentage 
in study SDAs), and 44 percent are receiving AFDC (the highest percentage in the study). 
Average participation in the program for youth in program year 1986 was 7 weeks, with many 
being served in the vocational exploration program. 

The SDA met ail its performance standards in program year 1986. 
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PCJTP staff condtict intake, eligibility detennination, and assessment of applicants. Hie 
background information form and informed consent form were completed as part of an 
assasment interview. During the assessment interview, the counselor determined whether 
further testing was required. !f not, as was often the case for those recommended for OJT, 
job search assistance, and vocational exploration, staff caUed MDRC for random assignment 
durmg the assessment interview and informed the client of the result at that time. If further 
testing was needed, as was possible for classroom occupational training, this additional 
assessment was completed before random assigssient occurred. 

The increasing difficulty the SDA faced in recruiting cUents because of the decline in 
the areas unemployment rate created issues in the implementation of the study. Early in 
random assignment, tiie SDA encountered serious problems recruiting youUi and meeting the 
required 40 percent of expenditiaes for services to youth. This was soon followed by problems 
recruitmg adults. To allow continuation of the study, tiie random assignment ratio was changed 
to ease tiie recruitment difficulties. Service providers also did less intensive testing of 
applicants, and some shifted to individualized open/entiy, open/exit services as opposed to 
traditional group training. 

The SDA did not meet two of its adult standards in program year 1987, in part because 
of the attention paid tiiroughout tiie year to the problems in youtii programs. As mentioned 
m Chapter 4, tiie state did not adopt adjustments in tiie performance standards suggested by 
tiie U.S. Department of Labor, and incentive funds to tiie SDA declined as a result 

Random assignment began in tiiis SDA in November 1987 and ended in September 
1989, witii the SDA meeting its sample target 
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APPENDIX C 

TABLES COMPARING PARTICIPATING SDAs WITH THE 
OTHER SDAs CONTACTED DURING SITE RECRUITMENT 
AND Vfmi THE POOL OF POTENTIAL SDAs FOR THE STUDY 
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TABLE C.l 



SELECTED CHARACTERISTICS AND PERFORMANCE STANDARDS OF SDAs 
CONTACTED DURING SITE RECRUITMENT, BY PARTICIPATION STATUS 



Characteristic/ 
Perfomance Standard 



Participating 
SDAs 



Non-Participating 
SDAs 



Total 



LOCAL ScHINS 

Unefflployircnt Rate in Program Year 1985 (%) 

Low (0% or more but less than 5V) 6.3 

Medium (5V or more but less than 8%) 50.0 

High (8V or more) 43.8 

Average SDA Unemploymeni Rate 

(% in program year 1985) 7.8 

Change in SDA Unemployment Rate from 1985 to 
1986 (%) 

Decreased more than 1 percentage point 18.8 

Decreased no more than 1 25.0 

Increased less than 1 or stayed the same 43.8 

Increased 1 or more 12.5 

Average Change in SDA Unemployment Rate 

from 1985 to 1985 (percentage points) 0.1 

Population Density (%) 

Less than 500 persons per square mile 75.0 

500 or more but less than 1,000 6.3 

1,000 or more but less than 5,000 6.3 

5,000 or more 12.5 

Average Population Density 

(persons per square mile) 1,245 
SIZE OF PROGRAM IH PROGRAM YEAR 1986 



18.8 
49.8 
31.5 



7.4 



8.9 
21.2 
41.5 
28.3 



-0.3 



66.7 
9.9 
13.1 
10.3 



1,172 



17.9 
49.8 
32.3 



7.4 



9.6 
21.5 
41.7 
27.2 



-0.3 



67.2 
9.6 
12.7 
10.5 



1,177* 



Tenninees from Title IIA (%) 
Less than 750 

750 or more but less 1,000 
1,000 or more but less than 1,500 
1,500 or more 

Average Tenninees from Title IIA 



50.1 
25.0 
12.5 
12.5 

957 



20.1 
18.8 
19.7 
41.3 



22.3 
19.2 
19.2 
39.3 



1,957 



1.888 
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TABLE C.l (continued) 



Characteristic/ 
Performance Standard 



CHARACTERISTICS OF PEOPLE SERVED UHDER 
TITLE IIA IN PR06RAN YEAR 1986 

White Terminees {%) 
Less than 25% 

25% or more but less than 50% 
50% or roore but less than 75% 
75% or more 

Average White Terminees {%) 

High School Dropouts {%) 
Less than 25% 

25% or more but less than 50% 
50% or more but less than 75% 
75% or more 

Average High School Dropouts {%) 

Adult High School Graduates {%) 
Less than 25% 

25% or more but 1 ess th?" 50% 
50% or more but less than 75% 
75% or more 

Average Adult High School Graduates {%) 

Welfare Recipient {%) 
Less than 25% 

25% or more but less than 50% 
50% or more but less than 75% 
75% or more 

Average Welfare Recipients {%) 
PERFORHAKCE STANDARDS* 

Adjusted'' Adult Entered Employment Rate {%)<= 
Less than 0 percentage points 
0 or more but less than 5 
5 or more but less than 10 
10 or more but less than 15 
15 or more 

Average Adjusted Adult Entered 
Employment Rate {%) 



Participating 
SOAs 



Non-Participating 
SDAs 



Total 



25.0 
12.5 
18.8 
43.8 

59.8 



56.3 
43.8 
0.0 
0.0 

25.5 



0.0 

0.0 
37.5 
62.5 

75.7 



56.3 
37.5 
6.3 
0.0 

26.9 



6.3 
0.0 
18.8 
37.5 
37.5 



16.9 
30.5 
21.1 
31.5 

54.9 



55.9 
43.7 
0.5 
0.0 

24.7 



0.0 
0.9 
49.8 
49.3 

74.8 



44.1 
47.9 
8.0 
0.0 

28.8 



8.S 
15.5 
24.9 
19.7 
31.0 



17.5 
29.3 
21.0 
32.3 

55.3 



55.9 
43.7 
0.4 
0.0 

24.8 



0.0 
0.9 
48.9 
50.2 

74.9 



45.0 
47.2 
7.9 
0.0 

28.8 



8.7 
14.4 
24.5 
21.0 
31.5 



10.7* 

(continued) 




TABLE C.l (continued) 



Characteristic/ 
Performance Standard 



Participating 
SDAs 



Non-Participating 
SDAs 



Total 



Adjusted Wage at Placement (V)*" 
Less than $0 

$0 or more but less than $1 
$1 or more but 1 ess than $2 
$2 or more 

Average Adjusted Wage at Placement ($) 

Adjusted Welfare Entered Employment Rate 
Less than 0 percentage points 
0 or more but less then 5 
5 or more but less than 10 
10 or more 

Average Adjusted Welfare Entered 
Employment Rate (%} 

Adjusted Average Cost per Entered 
Emf.l.iyment (V)' 
Less than -$2,500 

-$2,500 or more but less than -$1,000 
-$1,000 or more but less than $0 
$0 or more 



Average Adjusted Average Cost per Entered 
Employment ($) 

Adjusted Youth Entered Employment Rate (%)9 
Less than 0 percentage points 
0 or more but less than 5 
5 or more but less than 10 
10 or more 

Average Adjusted Youth Entered 
Employisent Rate (%) 



26.7 
6.7 
0.0 

66.7 

0.36 



0.0 
6.3 
25.0 
68.8 



18.8 



25.0 
50.0 
25.0 
0.0 



-1,595 



6.3 
31.3 
12.5 
50.0 



11.3 



9.6 
4.8 
7.7 
78.0 

0.46 



7.8 

17.6 
21.0 
53.7 



11.9 



15.0 
62.4 
17.4 
5.2 



-1,615 



16.4 
9.9 
15.1 
58.5 



13.5 



10.7 
■4.9 
7.1 

77.2 

0.45 



7.3 
16.7 
21.3 
54.8 



12.4* 



15.7 
61.6 
17.9 
4.8 



-1,613 



15.7 
11.4 
14.9 
57.9 



13.3 



(cciitinued) 
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TABLE C.l (continued) 



Characteristic/ Participating Non-Participating 

Perfonnance Standard SDAs SDAs Total 



LENGTH OF PROGRAM SERVICES UNDER TITLE IIA 
IN PROGRAH YEAR 1986 



-188- 



Adjusted Cost per Positive Termination 

Less than -$2,500 I2.5 ^-,5 jg 3 

-$2,500 or more but less than -$1,000 50.0 55.9 55*5 

-$1,000 or more but less than $0 31.3 23 5 24 0 

^^0^^^ 6.3 5!2 5:2 

-1,510 



Average Adjusted Cost per Positive 

Termination ($) .1,474 .j^gjj 



13.6 13.2 



Adjusted Youth Positive Termination 
Rate (Hy 

Less than 0 percentage points 6.3 

0 or more but less than 5 6.3 23 5 22*3 

5 orrwre but less than 10 37.5 24*4 25 3 

10 or more but less than 15 37.5 24 4 25 3 

12.5 I4!l u,o 

Average Adjusted Youth Positive 

Termination Rate (%) IC.4 



7.5 7.7 



Weeks of Enrollment for Adults (%) 

Less than 10 6.3 8.5 8 3 

10 or more but less than 20 43.8 60. i sg.'o 

20 or more but less than 30 43.8 26 f 27 9 

30 or more 6.3 a'/j a.B 

Average Weeks of Enrollment for Adults 18.3 ,17.2 

Weeks of Enrollment for Youth (V) 

Less than 10 6.3 75 y ^ 

10 or more but less than 20 37.5 6l!5 59.8 

20 or more but less than 30 50. 0 24 9 26 6 

^Oor^r^ 6.3 g/j ^-^^ 



Average Weeks of Enrollment for Youth 19.9 17.9 



17.3 



18.1 



Sample size 16 213 229 

(continued) 



TABLE C.l (continued) 



SOURCE: JTPA Annual Status Report (JASR) for program year 1986. 

NOTES: Due to missing data, the sample size for individual SDA characteristics varies from 
15 to 16 for participating SDAs and from 209 to 213 for non-participating SDAs. 

An F-test was applied to differences (for unweighted averages only) between 
the Itudy)'"^ """^ °^ potential SDAs (not including those that participated in 

"State performance standards for individual SDAs are set by governors and are 
intended to reflect local labor market conditions and characteristics of persons served. 
Adjusted adult entered employment rate is the SDA's actual adult entered 
s'imilarlj! P^-^^™"" '^"'^^'^'i- Other adjusted measures of perfomance are 

jThe percent of all adult teminees who entered employment. 
Average hourly wage of adults who entered employment at temination. 
The percent of all adult welfare recipient teminees who entered employment, 
employment «''P«''^itures for adults divided by the number of adults who entered 

^The percent of all youth teminees who entered employment. 

ten.inatiJ!'"U"''?:l':tTi!r °' '''' ' ""^"^^^^ 

«„ni ''^r""^ °l teminees who had a positive temination: either entered 

employment or met one of the employability enhancement definitions 
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TABLE C.2 



SELECTED CHARACTERISTICS AND PERFORMANCE STANDARDS OF SDAs 
CONTACTED DURING SITE RECRUITMENT. BY PARTICIPATION STATUS 
(WEIGHTED BY NUMBER OF TERHINEES) 



Characteristic/ 
Performance Standard 



Participating 
SDAs 



Non-Participating 
SOAs 



Total 



LOCAL SEHINS 

Unemployment Rate in Program Year 1986 (%) 
Low (0% or more but less than 5%) 4,2 
Medium (5V or more but less than 8%) 44,4 
High (8% or more) 51^5 

Average SDA Uncmpli)yment Rate 

{H in program year 1986) 8,1 

Change in SDA Unemployment Rate from 1985 to 
1986 (%) 

Decreased more than 1 percentage point 16.6 

Decreased no more than 1 15,6 

Increased less than 1 or stayed the same 53 ,4 

Increased 1 or more 14,3 

Average Change in SOA Unemployment Rate 

from 1985 to 1986 (percentage points) -O.l 

Population Density (%) 

Less than 500 persons per square mile 76.2 

500 or more but less than 1,000 5.6 

1,000 or more but less than 5.000 4.2 

5,000 or mere 14.0 

Average Population Density 

(persons per square mile) 1,258 

SIZE OF PROGRAM IN PROGRAM YEAR 1986 

Terminees from Title IIA (%) 
Less than 750 

750 or more but less 1,000 
1.000 or more but less than 1,500 
1,500 or more 

Average Teminees from Title IIA 1,371 



30.5 
23-2 
15.2 
31.2 



10.3 
53-6 
36.1 



7.7 



9.8 
19.1 
34.5 
36.8 



-0.3 



50,6 
10.8 
13.7 
24.8 



2,083 



5.6 
8.4 
12.4 
73.6 



10.0 
53.3 
36.7 



7.8 



10.0 
19.0 
35.2 
35.8 



-0.3 



Fi.5 
10.6 
13.4 
24.4 



2.054 



6.5 
8.9 
12.5 
72.1 



6.787 



6.595^ 



(continued) 
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TABLE C.2 (continued) 



Characteristic/ 
Performance Standard 



Participating 
SDAs 



Hon-Participating 
SDAs 



Total 



CHARACTERISTICS OF PEOPLE SERVED UNDER 
TITLE IIA IN PROSRAN YEAR 1986 

White Tenninees (%) 
Less than 25% 

25% or more but less than 50% 
50% or more but less than 75% 
75% or more 

Average White Tenninees {%) 

High School Dropouts (%) 
Less than 25% 

25% or more but less than 50% 
50% or more but less than 75% 
75% or more 

Average High School Dropouts (%) 

Adult High School Graduates (%} 
Less than 25% 

25% or more but less than 50% 
50% or more but less than 75% 
75% or more 

Average Adult High School Graduates (%) 

Welfare Recipients (%) 
Less than 25% 

25% or more but less than 50% 
50% or more but less than 75% 
75% or more 

Average Welfare Recipients {%) 

PERFORMANCE STANDARDS* 

Adjusted** Adult Entered Employment Rate (%)* 
Less than 0 percentage points 
0 or more but less than 5 
5 or more but less than 10 
10 or more but less than 15 
15 or more 

Average Adjusted Adult Entered 
Employment Rate (%) 



25.5 
9.7 
35.2 
29.5 

56.2 



42.2 
57.8 
0.0 
0.0 

27.6 



0.0 

0.0 
50.6 
49.4 

73.5 



64.4 
31.3 
4.4 
0.0 

24.4 



4.4 
0.0 
15.7 
45.8 
34.1 



14.9 



36.6 
27.0 
16.2 
20.1 

42.1 



46.3 
53.5 
0.2 
0.0 

26.8 



0.0 
0.4 
61.6 
38.0 

73.3 



47.9 
41.6 
10.5 
0.0 

28.6 



8.5 
12.4 
27.2 
25.8 
26.1 



10.7 



36.2 
26.4 
16.9 
20.5 

42.6 



46.2 
53.6 
0.2 
0.0 

26.8 



0.0 
0.4 
61.2 
36.4 

73.3 



48.5 
41.2 
10.3 
0.0 

28.5 



8.4 
12.0 
26.8 
26.5 
26.4 



10.8 



(continued) 
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TABLE C.2 (continued) 



Characteristic/ 
Performance Standard 



Participating 
SDAs 



Adjusted Wage at Placement 
Less than $0 

$0 or laore but less than $1 
$1 or iRore but less than $2 
$2 or more 

Average Adjusted Wage at Placetsent {$) 

Adjusted Welfare Entered Emplo^nt Rate {%)« 
Less than 0 percentage points 
0 or more but less than 5 
5 or more but less than 10 
10 or more 

Average Adjusted Welfare Entered 
Employment Rate (*) 

Adjusted Average Cost per Entered 
Employment (%)' 
Less than -$2,500 

-$2,500 or more but less than -$1,000 
-$1,000 or more but less than $0 
$0 or more 



20.7 
7.3 
0.0 

72.0 

0.42 



0.0 
5.6 
15.8 
78.7 



19.9 



21.1 
59.8 
19.2 
0.0 



Average Adjusted Average Cost per 
Entered Employment ($) 

Adjusted Youth Entered Employment Rate (V)S 
Less than 0 percentage points 
0 or more but less than 5 
5 or more but less than 10 
10 or more 

Average Adjusted Youth Entered 
Employment Rate (%) 



-1,589 



2.7 
23.5 
13.4 
60.4 



13.9 



Non-Participating 
SDAs 



8.9 
8.4 
10.1 
72.6 

0.39 



7.0 
14.6 
31.1 
47.1 



11.7 



21.9 
60.8 
13.1 
4.2 



-1,776 



14.1 
8.9 
13.4 
63.6 



13.4 



9.3 
" 8.3 

9.8 
72.6 

0.39 



6.7 
14.5 
30.5 
48.3 



11.9** 



21.8 
60.8 
13.3 
4.0 



1,769 



13.7 
9.4 
13.4 
63.5 



13.4 



(continued} 
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TABLE C.2 (continued) 



Characteristic/ 
Performance Standard 



Participating 
SDAs 



Non-Participating 
SDAs 



Adjusted Cost per Positive Termination {%)^ 
Less than -$2,500 

-$2,500 or more but less than -$1,000 
-$1,000 or more but less than $0 
$0 or more 



8.9 
42.3 
44.3 

4.4 



Average Adjusted Cost per Positive 

Termination {$) -1,275 



Adjusted Youth Positive Termination Rate (%)^" 
Less than 0 percentage points 
0 or more but less than 5 
5 or more but less than 10 
10 or njore but less than 15 
15 or more 

Average Adjusted Youth Positive 
Termination Rate (%] 

LEHGTK OF PROGRAM SERVICES UNDER TITLE IIA 
IN PROGRAM YEAR 1985 

Weeks of Enrollment for Adults {%) 
Less than 10 

10 or more but less than 20 
20 or more but less than 30 
30 or more 

Average Weeks of Enrollment 
for Adults 

Weeks of Enrollment for Youth {%) 
Less than 10 

10 or more but less than 20 
20 or more but less than 30 
30 or more 

Average Weeks of Enrollment 
for Youth 



4.4 
4.1 
46.8 
31.8 
13.0 



10.9 



5.0 

57.9 
32.7 
4.4 



17.0 



4.2 

32.4 
56.9 
6.5 



19.8 



Total 



28.3 
53.0 
15.2 
3.5 



-1,735 



9.1 
31.4 
27.1 
17.1 
15.3 



7.9 



13.6 
66.7 
16.5 
3.2 



15.3 



7.4 
68.8 
18.9 

4.9 



16.8 



27.6 
■52.6 
16.3 
3.5 



-1,719 



8.9 
30.4 
27.8 
17.6 
15.2 



8.0 



Sample Size 



16 



213 



13.3 
56.4 
17.1 
3.2 



15.4 



7.3 
67.5 
20.2 

5.0 



16.9 



229 



220 
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TABLE C.2 (continued) 
SOURCE: JTPA Annual Status Rei»rt (JASR) for program year 1986. 

HOTES: Weighted numbers are calculated by weighting SDAs by a measure of the number of 
people served. 

,c * ,c ^° "i"'"^ ^o*" individual SDA characteristics varies from 

15 to 16 for participating SDAs and from 209 to 213 for non-participating SDAs 

. A *PP^i«'' ^» differences (for averages only) between participating SDAs 

and the pool of potential SDAs (not including those that participated in the study). 

Perfo"™"" standards for individual SDAs are set by governors and are 
intendea Jo reflect local labor market conditions and characteristics of persons served. 
Adjusted adult entered employment rate is the SDA's actual adult entered 

c^rSed stii^;?;! ''''^ -^^'"^^'^^ p^'^^™-- 

jThe percent of all adult terminees who entered employment. 
Average hourly wage of adults who entered employment at termination. 
The percent of all adult welfare recipient terainees who entered employment, 
employment "P^"''^^"'-" for adults divided by the number of adults who entered 

9The percent of all youth terminees who entered employment. 
^°^«l/«Pf ''itures for youth divided by the number of youth who had a positive 
termination. (See footnote i.) o poimve 

^The percent of all youth terminees who had a positive cennination: either entered 
employment or met one of the employability enhancement definittons. 



ERIC 



-19A- 



TABLE C.3 

ESTIMATED REGRESSION COEFFICIENTS FOR 
THE PROBABILITY OF AN SDAs AGREEING TO PARTICIPATE 
IN THE NATIONAL JTPA STUDY 



Regressor or Statistic 



Coefficient 



LOCAL SETTING 

Region 
Northeast 
South 
Midwest 
West 

SDA Unemployment Rate (program year 1986) 

Change in SDA Unemployment 

Rate from 1985 to 1986 (percentage points) 

Population Density (persons per square mile) 

SIZE OF PROGRAM IN PROGRAM YEAR 1986 

Terminees from Title IIA 

CHARACTERISTICS OF PEOPLE SERVED 
UNDER TITLE IIA IN PROGRAM YEAR 1985 

Percent of Tenninees Who Are White 

Percent of Terminees Who Are High School Dropouts 

Percent of Tenninees Who Are Welfare Recipients 

LENGTH OF PROGRAM SERVICES UNDER 
TITLE IIA IN PROGRAM YEAR 1985 

Average Weeks of Enrollment for Adults 

Average Weeks of Enrollment for Youth 

ADULTS 

Entered Employment Rate* 
Adjusted^ Entered Employment Rate 

Welfare Entered Employment Rate^ 
Adjusted Welfare Entered Employment Rate 

Wage at Placement^ 
Adjusted Wage at Placement 



0.004624 

0.169530* 
0.021490 

0.000002 



0.000294* 
0.000036 

-0.000007 



0.068730 
0.154020 
-0.096730 



-0.003367 
0.000570 



0.000067 
-0.000022 

-0.000004 
0.000039 

-0.000988 
0.000665 



(continued) 
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TABLE C.3 (continued) 



Regressor or Statistic 



Coefficient 



Cost par Entered Employment® 
Adjusted Cost per Entered foploynent 

YOUTH 

Positive Tenni nation Rate^ 
Adjusted Positive Termination Rate 

Entered Employment RateS 
Adjusted Entered Employment Rate 

Cost per Positive Termination*' 
Adjusted Cost per Positive Tennination 



0.CO0O62 
•0.000025 



0.000003 
0.000022 

-0.000015 
-0.000013 

0.000029 
-0. 000022 



Number of Observations 
Number of Participating SDAs 
Number of Other SDAs in Pool 
Degrees of Freedom for Error 
Error Mean Square 
R Square 

Mean of Dependent Variable 
F Statistic 

P Value of F Statistic 



228 
16 

212 

201 
0.06 
0.14 
0.070 
1.23 
0.21 



Sample Size 



228 



SOURCE: 



OTPA Annual Status Report (JASR) for program year 1986. 



tTp'inicija'^eiTz^ :t":^';:e:" "^""^^ "-^^^ ^^-^ 

levels areVn'S^vltll.*'' Tn '^^^'^ '° coefficient estimate. Statistical significance 

levels are indicated as: * « 10 percent; ** « 5 percent; and *** - 1 percent 

estimate, L'T' "^V" f "^"^^'^ '"^ ^^e probability of obtaining these coefficient 
estimates if the true chance of an SDA participating in the study did not vary with anv 

ch."r!LT"I-'- J'"- '''''' " ^"^"^ '^^rll are differences ^ 

characteristics between participating SDAs and non-participating SDaV. 

Jhe percent of all adult terminees who entered employment. 
«,-„,.c adult entered employment rate is the SDA's actual entered employment rate 

minus Its perfo™»nce standard, other adjusted measures of perfomance are calculated Slarfy! 
The percent of all adult welfare recipient teminees who entered employment. 
Average hourly wage of adults who entered employment at tennination. 
Total expenditures for adults divided by the number of adults who entered employment. 
The percent of all youth terminees who hod a positive tennination: either entered 
employment or met one of the employability enhancement definitions. 

SThe percent of all youth teminees who entered employment. 
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TABLE C.4 



SELECTED CHAR'aCTERISTICS AND PERFORMANCE STANDARDS 
OF PARTICIPATING SDAs AND THE POOL 
OF POTENTIAL SDAs FOR THE STUDY 



Characteristic/ 
Psrfonnance Standard 



Non- Pool of 

Participating Participating Participating 
SDAs SDAs SDAs 



LOCAL SETTIN5 

Unemployment Rate in Program Year 1986 (%) 

Low (0% or more but less than 5%) 6.3 

Medium (5% or more but less than 8%) 50.0 

High (8* or more) 43.8 

Average SDA Unemployment Rate 

(% in program year 1986) 7.8 

Change in SDA Unemployment Rate from 1985 to 
1986 (%} 

Decreased more than 1 percentage point I8.8 

Decreased no more than 1 25.0 

Increased less than 1 or stayed the same 43.8 

Increased 1 or more 12.5 

Average Change in SDA Unemployment Rate from 

1985 to 1986 (percentage points) 0.1 

Population Density {%) 

Less than 500 persons per square mile 75.0 

500 or nwre but less than 1,000 6.3 

1,000 or more but less than 5,000 6.3 

5,000 or more 12.5 

Average Population Density 

(persons per square mile) 1,245 
SIZE OF PROGRAM IN PROGRAM YEAR 1986 



14.5 
45.5 
40.0 



8.0 



15.3 
22.5 
38.2 
24.0 



-0.1 



74.8 
8.7 

11.2 
5.4 



772 



14.3 
45.6 
40.1 



8.0 



15.4 
22.6 
38.4 

22.5 



-0.1 



74.8 
8.6 

11.0 
5.6 



787 



Tenninees from Title IIA (%) 
Less than 750 

750 or more but less than 1,000 
1,000 or more but less than 1,500 
1,500 or more 



50.1 
25.0 
12.5 
12.5 



26.9 
20.1 
23.8 
29.3 



27.6 
20.2 
23.4 
28.7 



Average Teminees from Title IIA 



957 



1,561 



1.542 
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TABLE C.4 (continued) 



Characteristic/ 
Perfonwnce Standard 



CHARACTERISTICS OF PEOPU SERVSO UNDER 
TITLE IIA IN PROGRAM YEAR 19ae 

White Terminees {%) 
Less than 25% 

25% or more but less than 50% 
50% or more but less than 75% 
75% or more 

Average White Terminees (%) 

High School Dropouts (%) 
Less than 25% 

25% or more but less than 50% 
50% or more but less than 75% 
75% or more 

Average High School Dropouts (%) 

Adult High School Gradu«.ces (%) 
Less than 25% 

25% or more but less than 50% 

50% or more but less than 75% 
75% or more 

Average Adult High School Graduates (%) 

Welfare Recipients (%) 
Less than 25% 

25% or more but less than 50% 
50% or more but less than 75% 
75% or more 

Average Welfare Recipients (%) 

PERFORHAHCE STANDARDS* 

Adjusted'^ Adult Entered Employment Rate (%)^ 
Less than 0 percentage points 
0 or more but less than 5 
5 or mor^ but less than 10 
10 or more but less than 15 
15 or more 

Average Adjusted Adult Entered 
Employment Rate (%) 



Non- Pool of 

Participating Participating Participating 



SDAs SDAs SDAs 



25.0 13.5 13.9 

12.5 25.6 25.2 

18.8 21.9 21.8 

43.8 38.9 39.1 

59.8 59.9 59.9 



56.3 54.9 55.0 

43.8 44.7 44.6 

0.0 0.4 0.4 

0.0 0.0 0.0 

25.5 24.7 24.8 



0.0 0.0 0.0 

0.0 1.6 1.6 

37.5 49.0 48.6 

62.5 49.4 49.8 

75.7 74.2 74.3 



56.3 44.5 44.8 

37.5 46.1 45.8 

6.3 9.2 9.1 

0.0 0.2 0.2 

26.9 28.8 28.8 



6-3 8.4 8.4 

0-0 15.0 14.5 

18.8 24.8 24.6 

37.5 21.8 22.0 

37.5 30.3 30.6 



14.3 10.5 10. 6* 
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TABLE C.4 (continued) 



Hon- Pool of 

Characteristic/ Participating Participating Participating 

Perfonnance Standard SDAs SDAs SD/te 



Adjusted Wage at Placement (%)^ 
Less than $0 

$0 or more but less than $1 
$1 or iTJore but less than $2 
$2 or more 

Average Adjusted Wage at Placement ($) 

Adjusted Welfare Entered Employment Rate (V) 
Less than 0 percentage points 
0 or more but less than 5 
5 or more but less than 10 
10 or !!sore 

Average Adjusted Welfare Entered 
Employn>ent Rate (%) 

Adjusted Average Cost per Entered 
Employment (%)* 
Less than -$2,500 

-$2,500 or more but less than -$1,000 
•$1,000 or more but less than $0 
$0 or more 

Average Adjusted Average Cost per Entered 
Employment ($) 

Adjusttxl Youth Entered Employment 
Rate (U)^ 

Less than 0 percentage points 
0 or more but less than 5 
5 or more but less than 10 
10 or more 

Average Adjusted Youth Entered 
Employment Rate (%) 



26.7 12,7 13.1 

6.7 6,6 6.6 

0.0 9.5 9,3 

66.7 71,2 71,0 

0.36 0.41 0.40 



0.0 7,0 6.7 

6*3 17.4 17.0 

25.0 20,2 20,3 

68.8 55,4 55.9 



18.8 11.9 12.1** 



25.0 15.2 15.5 

50.0 57,4 57.1 

25,0 20.9 21.0 

0.0 6.6 6.3 



-1,595 -1,511 -l,ol4 



6.3 17,0 16.6 

31.3 13.0 13.6 

12.5 15.9 15.8 

50.0 54.1 54.0 



11-3 12.2 12.2 
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TABLE C.4 (continued) 



'""S'""' P.rticip«,„5 P„t,cip..i„, 
. 5UAS SDAs SDAs 



Adjusted Cost per Positive Termination {%)^^ 

Less than -$2,500 12 5 13 3 

-$2,500 or more but less than -$1.000 so.'o * 

-$1,030 or more but less than $0 31 ^3 
$0 or tnorc 



6.3 4.7 



30 or more 



n ci 



54.3 54.2 
27.7 27.8 



4.8 



Average Adjusted Cost per Positive 

Temination ($) , , ..^ 

-1.474 -1,448 -1,445* 

Adjusted Youth Positive Termination Rate 
Less than 0 percentage points 6 3 ,c . 

0 or more but less than 5 6*3 23 2 'f 

5 or more but less than 10 37*5 ^'J 

10 or more but less than 15 37's , r:*! 

15 or more iJ'! fl.l 21.6 

12.5 16.2 26.1 

Average Adjusted Youth Positive 

Temination Rate {%) 10 4 ^ ^ 

LENGTH OF PROGRAM SERVICES UNDER TITLE IIA 
IN PROGRAM YEAR 1986 

Weeks of Enrollment for Adults (%) 

Less than 10 6 3 6 1 

10 or more but less than 20 43*8 s^'i cS'f 

20 more more but less than 30 ,/« 



43.8 31.8 32.1 
6.3 8.0 7.9 

Avcnige Weeks of Enrollment for Adults I8.3 is. 3 is s* 

Weeks of Enrollment for Youth (%) 
Less than 10 6345 

10 or more but less than 20 37*5 5, '5 

20 or more but less than 30 so.'o 33.1 34 3 

30 or more , ^ f , ^^-3 

6.3 8.2 8.1 

Average Weeks of Enrollment fo r Youth 19.9 jg 4 

Sample Size Ig 



19.4 



488 504 
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TABLE C.4 (continued) 



SOURCE: JTPA Annual Status Report (JASR) for program year 1986. 

NOTES: Due to missing data, the seunple size for individual SDA characteristics varies from IS 
to 16 for participating SDAs and from 48? to 504 for the pool of potential SDAs. 

An F-test was applied to differences (for unweighted averages only) between 
participating SDAs and the pool of potential SDAs (not including those that participated in the 
study) • 

^State performance standards for individual SDAs are set by governors and are intended 
to reflect local labor market conditions and characteristics of persons served. 

Adjusted adult entered employment rate is the SDA's actual adult entered employment 
similarly^ standard. Other adjusted measures of performance are calculated 

^The percent of all adult teminees who entered employment. 

^Average hourly wage of adults who entered employment at termination. 

^The percent of all adult welfare recipient terminees who entered employment. 

Total expenditures for adults divided by the number of adults who entered employment. 
^The percent of all youth teminees who entered employment. 

Total expenditures for youth divided by the number of youth who had a positive 
termination. (See footnote i.) 

The percent of all youth teminees who had a positive temination: either entered 
employment or net one of the employability enhancement definitions. 
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TABLE C.5 



SELECTED CHARACTERISTICS AND PERFORKAHCE STANDARDS 
OF PARTICIPATING SDAs AND THE POOL 
OF POTENTIAL SDAs FOR THE STUDY 
(WEIGHTED BY NUMBER OF TERMINEES) 



Characteristic/ 
Performance Standard 



Participating 
SDAs 



Non-Participsting 
SDAs 



Pool of 
Participating 
SDAs 



LOCAL SETTING 

Unemployment Rate in Program Year 1986 (%) 
Low (0% or more but less than 5%) 
Medium {5% or more but less than 8%) 
High (8V or more) 

Average SDA Unemployment Rate 
(% in program year 1986) 

Change in SDA Unemployment Rate from 1985 
to 1986 (*) 

Decreased more than 1 percentage point 

Decreased no more than 1 

Increased less than 1 or stayed the same 

Increased 1 or more 

Average Change in SDA Unemployment Rate 
from 1985 to 1986 (percentage points) 

Population Density (%) 
Less than 500 persons per square mile 
500 or more but less than 1,000 
1,000 or more but less than 5,000 
5,000 or more 

Average Population Density 
(persons per square mile) 

SIZE OF PROGRAM IN PROGRAM YEAR 1986 

Temiinees from Title IIA (V) 
Less than 750 

750 0' ore but less than 1,000 
1,000 jr more but less than 1,500 
1,500 or more 



4.2 
44.4 
51.5 



8.1 



0.0 
16.6 
15.6 
67.7 



-0.1 



76.2 
5.6 
4.2 

14.0 



1,258 



30.5 
23.2 
15.2 
31.2 



11.4 
45.6 
43.0 



8.0 



6.4 
8.0 
21.7 
64.0 



-0.1 



63.8 
9.5 
12.4 
14.4 



1.366 



10.2 
11.2 
18.7 
59.9 



11.2 
45.6 
43.1 



8.1 



6.3 
8.2 
21.5 
64.1 



-0.1 



64.0 
9.4 
12.2 
14.4 



1,364 



10.6 
11.4 
18.6 
59.4 



Average Terminees from Title IIA 



1,371 



5,109 



5,035 



(continued) 
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TABLE C*5 (continued) 



Pool of 

Characteristic/ Participating Non-Participating Participating 

Performance Standard SDAs SDAs SDAs 



CHARACTERISTICS OF PEOPLE SERVED UNDER TITLE IIA 
IN PRQGRAH YEAR 1986 



White Terminees (%} 
Less than 25% 

25% or more but less than 50% 
50% or more but less than 75% 
75% or more 

Average White Terminees (%) 

High School Dropouts (%} 
Less than 25% 

25% or more but less than 50% 
50% or more but less than 75% 
75% or more 

Average High School Dropouts (%) 

Adult High School Graduates (%) 
Less than 25% 

25% or more but less than 50% 
50% or more but less than 75% 
75% or more 

Average Adult High School Graduates (%) 

Welfare Recipients (%) 
Less than 25% 

25% or more but less than 50% 
50% or more but less than 75% 
75% or more 

Average Welfare Recipients (%) 

PERFORMANCE STANDARDS* 

Adjusted*^ Adult Entered Employment Rate (%) 
Less than 0 percentage points 
0 or more but less than 5 
5 or more but less than 10 
10 or more but less than 15 
15 or more 

Average Adjusted Adult Entered 
Employment Rate (%) 



25.5 26.9 26.9 

9.7 26.0 25.7 

35.2 17.6 17.9 

29.5 29.4 29.4 

56.2 49.6 49.7 

42.2 48.8 48.7 
57.8 48.5 48.7 

0.0 2.7 2,6 

0.0 0.0 0.0 

27.6 26.9 26.9 

0.0 0.0 0.0 

0.0 3.7 3.6 

50.6 54.6 54.5 

49.4 41.7 41.9 

73.5 72.5 72.5 

64.4 50.0 50.3 

31.3 39.7 39.5 
4.4 10.2 10.1 
0.0 0.1 0.1 

24.4 27.8 27.8 



^^.4 8.5 8.5 

0.0 16.6 16.3 

15.7 25.1 25.0 

45.8 22.4 22.9 
34.1 27.2 27.4 

14.9 10.2 10.3* 
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TABLE C.5 (continued) 



Characteristic/ 
Performance Standard 



Participating 
SDAs 



Hon-Participating 
SDAs 



Pool of 
Participating 
SDAs 



Adjusted Wage at Placement 
Less than $0 

$0 or B»re but less than $1 
$1 or more but less than $2 
$2 or more 

Average Adjusted Wage at Placement ($) 

Adjusted Welfare Entered Employment Rate (%)* 
Less than 0 percentage points 
0 or more but less than 5 
5 or more but less than 10 
10 or more 

Average Adjusted Welfare Entered 
Employment Rate (%) 

Adjusted Average Cost per Entered 
Employment (%)' 
Less than -$2,500 

-$2,500 or more but less than -$1,000 
-$1,000 or more but less than $0 
$0 or more 

Average Adjusted Average Cost per Entered 
Employment ($) 

Adjusted Youth Entered Employment 
Rate (%)9 

Less than 0 percentage points 

0 or more but less than 5 

5 or more but less than 10 

10 or more 

Average Adjusted Youth Entered 
Employment Rate (%) 



20.7 
7.3 
0.0 

72,0 

0,42 



0.0 
5.6 
15.8 
78.7 



19.9 



21.1 
59.8 
19.2 
0.0 



-1,589 



2.7 
23.5 
13.4 
60.4 



13.9 



11.5 
10.3 
9.8 
68.4 

0.36 



8.7 
15.0 
26.4 
49.9 



11.4 



18.7 
57.9 
15.9 
7.5 



-1,575 



15.1 
13.7 
14.3 
56.9 



12.1 



11.7 
10.3 
9.6 
68.5 

0.36 



8.6 
14.8 
26.2 
50.4 



!1.6** 



18.7 
57.9 
15.9 
7.4 



-1,575 



14.8 
J3.9 
14.3 
57.0 



12.1 
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TABLE C.5 (continued) 



Characteristic/ 
Performance Standard 



Average Adjusted Cost psv Positive 
Tenni nation ($) 

Adjusted Youth Positive Termination Rate (*)^ 
Less than 0 percentage points 
0 or more but less than 5 
5 or n»re but less than 10 
10 or more but less than 15 
15 or more 

Average Adjusted Youth Positive 
Tenni nation Rate (%) 

LEHGTH OF PROGRAM SERVICES UNDER TITLE IIA 
IN PROGRAM YEAR 1986 

Weeks of Enrollment for Adults (%) 
Less than 10 

10 or more but less than 20 
20 more more but less than 30 
30 or more 

Average Weeks of Enrollment for Adults 

Weeks of Enrollment for Youth (%) 
Less than 10 

10 or more but less than 20 
20 or more but less than 30 
30 or more 

Average Weeks of Enrollment for Youth 



Participating 
SDAs 



Non- 



Adjusted Cost per Positive Termination (%)*^ 
Less than -$2,500 

-$2,500 or more but less than -$1,000 
-$1,000 or more but less than $0 
$0 or irore 



8.9 
42.3 
44.3 

4.4 



-1,275 



4.4 
4.1 
46.8 
31.8 
13.0 



10.9 



5.0 
57.9 
32.7 

4.4 

17.0 



4.2 
32.4 
56.9 

6.5 

19.8 



•Participating 
SDAs 



Pool of 
Participating 
SDAs 



20.5 
54.7 
21.3 
3.5 



-1,612 



12.0 
28.3 
25.1 
16.4 
18.2 



7.4 



9.3 
58.9 
23.6 

8.2 

17.6 



5.0 
56.9 
30.9 

7.2 

18.6 



20.3 
54.5 
21.7 
3.5 



-1,605 



11.9 
27.8 
25.5 
16.7 
18.1 



7.5 



9.2 
58.9 
23.8 

8.1 

17.6 



5.0 
56.4 
31.4 

7.2 

18.6 



Sample Size 



16 



488 



504 
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TOBLE C.5 (continued) 



SOURCE: OTPA Annuil Status Report (OASR) for progren year 1986. 

KOIB,^ Weighted n,»4ers .re cal«l.t«l by ..ijhting sDAs by a «asur. of the number of people 

for paj^pajts'-r:;; »:oi« z rx^^^- - - » 
^ poo, s;K';„it'?„2i::?„T?sL'^n^^^^ ?;r=s;r-'"' ^"^^ 

d*"!. : ? tennmees who entered emplojinent. 

£S r 7"^^^^^^^^^^^^ rr:5tp,o^nt. 

te™„n.t,-.;"jlerS«:" )" °' >*> X" • p°--««' 

.„o,«.n:^r ^r::; oi 2;iiari;r:n;:^r!c:i: s:?!j;;?or""'°"= 
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TABLE C.6 



ESTIHATED REGRESSIOH COEFFICIENTS FOR 
THE PROBABILITY OF AH SDA IN THE POOL OF POTENTIAL STUDY SITES 
PARTICIPATING IN THE STUDY 



Regressor or Statistic 



Coefficient 



LOCAL SEHINS 

Region 
Northeast 
South 
Hidwest 
Uest 

SDA Unemployment Rate (prograin year 1986) 

Change in SDA Unemployment 

Rate from 1985 to 1986 (percentage points) 

Population Density (persons per square mile) 

SIZE OF PROSRAH IN PROGRAM YEAR 1986 

Terminees from Title IIA 

CHARACTERISTICS OF PEOPLE SERVED 
UNDER TITLE IIA IN PROGRAM YEAR 1986 

Percent of Terminees Who Are White 

Percent of Terminees Who Are High School Dropouts 

Percent of Terminees Who Are Welfare Recipients 

LENGTH OF PROGRAM SERVICES UNDER 
TITLE IIA IN PROGRAM YEAR 1986 

Average Weeks of Enrollment for Adults 

Average Weeks of Enrollment for Youth 

ADULTS 

Entered Employment Rate* 
Adjusted*^ Entered Employment Rate 

Welfare Entered Employment Rate^ 
Adjusted Welfare Entered Employment Rate 

Wage at Placement^ 
Adjusted Wage at Placement 



0.034520 

0.083940** 
0.052890* 

*0. 000036 



0.000087 
0«000079 

-0.000006 



0.012420 
0.101760 
-0.058960 



-e.O01O36 
-O.O0U526 



0.000048 
-0.000044 

-0.000006 
0.000030 

-0.000516 
0.000321 
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TABLE C.6 (continued} 



Regressor or Statistic 



Coefficient 



Cost per Entered Bnplojreent® 
Adjusted Cost per Entered Bnployment 

YOUTH 

Positive Termination Rate^ 
Adjusted Positive Termination Rate 

Entered Enqjloyment RateS 
Adjusted Entered Employment Rate 

Cost per Positive Teraination^ 
Adjusted Cost per Positive Termination 



0.000016 
-0.000012 



-0.000019 
0.000033 

-0.000014 
-0.000002 

0.000015 
-0.000008 



Number of Observations 433 
Number of Participating SDAs 15 
Number of Other SDAs in Pool 
Degrees of Freedom for Error 
Error Hean Square 
R Square 

Kean of Dependent Variable 
F Statistic 

P Value of F Statistic 



467 
456 
0.03 
0.05 
0.033 
1.01 
0.45 



Sample Size ^2 

SOURCE: JTPA Annual Status Report (JASR) for program year 1986. 

te^nfitiv J^V'T"""^ "fu"""* '■«9'-«sion equation was unity for an SDA that agreed 

to participate and zero otherwise. 

A two-tailed t-test was applied to each coefficient estimate. Statistical significance 
levels are indicated as; * « 10 percent; - . 5 percent; and « l percent. 3"'-'""" 
estimate. ^/tU^J" F statistic is the probability of obtaining these coefficient 

estimates If the true chance of an SDA participating in the study did not vary with any 

ctr^tt'i-I « J '''^-"^ ' ''''' " '° -'^^^-^ differences i 

characteristics between participating SDAs and other SDAs in the pool of potential study sites 

Jhe percent of all adult tenninees who entered employment. 

Adjusted adult entered employment rate is the SDA's actual entered employment rate 
minus Its iHsrformance standard. Other adjusted measures of perfonnance are calculated Vimila"y! 
The percent of all adult welfare recipient teminees who entered employment. 
^Average hourly wage of adults who entered employment at tennination. 
Total expenditures for adults divided by the number of adults who entered employment 
The percent of all youth tenninees who had a positive tennination: either entered 
employment or met one of the employability enhancement definitions. 

SThe percent of all youth tenninees who entered employment. 

t.™,- "^"^/I expenditures for youth divided by the number of youth who had a positive 
tennination. (See footnote f.) a posmve 
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